CS 112 University of San Francisco Java Lexer Analyzer Computer Science Task

User Generated

frerani143

Programming

CS 112

University of San Francisco

CS

Description

Our program will have a Lexer class and a Token class. The Lexer provided in the starter code is incomplete. You can use the Token class as is.

  • The class Lexer should provide the following methods provided:   
    • public constructor which takes a file name as a parameter
    • private getInput which reads the data from the file into a String.
  • You’ll need to add the following methods to Lexer:
    • public getNextToken which returns a Token object.
    • private functions for getIdentifier and getInteger. These helper functions should handle extracting the rest of an identifier/integer once one is identified.
    • A getAllTokens which calls getNextToken within a loop and returns a list of Tokens.
  • The class Token should have two data members, type and value, both of type String, along with a constructor which takes two parameters, and a toString method which prints a token’s type and value.

This is the starter code and also uses the Token Class as well below 

import java.io.IOException;import java.nio.file.Files;import java.nio.file.Path;import java.nio.file.Paths;import java.util.ArrayList;import java.util.Scanner; /** * Class to build an array of Tokens from an input file * @author rena * @see Token * @see Parser */public class Lexer { //The class Token should have two data members, type and value, both of type String String type; //data member String value; //data member  String buffer; int index = 0; public static final String INTTOKEN="INT"; public static final String IDTOKEN="ID"; public static final String ASSMTTOKEN="ASSMT"; public static final String PLUSTOKEN="PLUS"; public static final String EOFTOKEN="EOF";  /**     * call getInput to get the file data into our buffer     * @param fileName the file we open */ public Lexer(String fileName) {         getInput(fileName);    }  public getNextToken(){ # no parameters     } /*private functions for getIdentifier and getInteger. These helper functionsshould handle extracting the rest of an identifier/integer once one is identified. */ private getIdentifier(){ # no parameters     }  private getInteger(){ # no parameter     }  //A getAllTokens which calls getNextToken within a loop and returns a list of Tokens. public getAllTokens(){ # no parameters // // call getNextToken    }

/**     * Reads given file into the data member buffer     * @param fileName name of file to parse */ private void getInput(String fileName)  { try { Path filePath = Paths.get(fileName); byte[] allBytes = Files.readAllBytes(filePath);            buffer = new String (allBytes);        } catch (IOException e) { System.out.println ("You did not enter a valid file name in the run arguments."); System.out.println ("Please enter a string to be parsed:"); Scanner scanner = new Scanner(System.in);            buffer=scanner.nextLine();        }    }  /**     * Return all the token in the file     * @return ArrayList of Token */  public ArrayList<Token> getAllTokens(){ //TODO: place your code here for lexing file return new ArrayList<Token>(); // don't forget to change the return statement    }     /**     * Before your run this starter code     * Select Run | Edit Configurations from the main menu.     * In Program arguments add the name of file you want to test (e.g., test.txt)     * @param args args[0] */ public static void main(String[] args) { String fileName=""; if (args.length==0) { System.out.println("You can test a different file by adding as an argument"); System.out.println("See comment above main"); System.out.println("For this run, test.txt used");            fileName="test.txt";        } else {             fileName=args[0];        } Lexer lexer = new Lexer(fileName); // just print out the text from the file System.out.println(lexer.buffer); // here is where you'll call getAllTokens     }}

public class Token {
   public String type;
   public String value;


   public Token(String type, String value) {
       this.type=type;
       this.value=value;
   }


   public String toString(){

       return type+" "+value;
   }


}

HERE IS OTHER INFO ABOUT THIS ASSIGNMENT :A Lexical Analyzer (or Lexer for short) performs the task of finding the tokens in a large set of text.A token is a sequence of characters that form a single entity in a language. For an English sentence, each word is a token. For the SIMPLE coding language, there are a number of token types, including Identifier, AssignOp, Plus, and Integer. Your program will have a Token class that identifies two parts of a token, its type and its value.For instance, given the text “x23=32”, a Lexer would identify three tokens: an Identifier with value “x23”, an AssignOp with value “=” and an Integer with value “32”.Your Lexer will take a text file as input and create a list of the tokens  contained in the file. The Lexer will identify the following token types:Identifier: starts with a letter followed by 0 or more letters and digits.Integer: a sequence of digitsAssignOp: a single ‘=’PlusOp: a single ‘+’UnknownOp: e.g., ‘$’EndOfFile: for every file, the last token generated by the Lexer will be this one.If your Lexer was provided the file:aa=34bb=45+6It will generate the following Tokens: Token Type  Token ValueIdentifier         aaAssignOp        =Integer           34Identifier         bbAssignOp       =Integer           45PlusOp           +Integer            6EndOfFile        -

Unformatted Attachment Preview

x32 = 83+ yzu rt=2 1 32=54 1 xyz = 33 zz12=99+88+xyz 1 x = 32 32 1
Purchase answer to see full attachment
User generated content is uploaded by users for the purposes of learning and should be used following Studypool's honor code & terms of service.

Explanation & Answer

Hello,Uploading the java files and te...


Anonymous
This is great! Exactly what I wanted.

Studypool
4.7
Trustpilot
4.5
Sitejabber
4.4

Related Tags