Description
Our program will have a Lexer class and a Token class. The Lexer provided in the starter code is incomplete. You can use the Token class as is.
- The class Lexer should provide the following methods provided:
- public constructor which takes a file name as a parameter
- private getInput which reads the data from the file into a String.
- You’ll need to add the following methods to Lexer:
- public getNextToken which returns a Token object.
- private functions for getIdentifier and getInteger. These helper functions should handle extracting the rest of an identifier/integer once one is identified.
- A getAllTokens which calls getNextToken within a loop and returns a list of Tokens.
- The class Token should have two data members, type and value, both of type String, along with a constructor which takes two parameters, and a toString method which prints a token’s type and value.
This is the starter code and also uses the Token Class as well below
import java.io.IOException;import java.nio.file.Files;import java.nio.file.Path;import java.nio.file.Paths;import java.util.ArrayList;import java.util.Scanner; /** * Class to build an array of Tokens from an input file * @author rena * @see Token * @see Parser */public class Lexer { //The class Token should have two data members, type and value, both of type String String type; //data member String value; //data member String buffer; int index = 0; public static final String INTTOKEN="INT"; public static final String IDTOKEN="ID"; public static final String ASSMTTOKEN="ASSMT"; public static final String PLUSTOKEN="PLUS"; public static final String EOFTOKEN="EOF"; /** * call getInput to get the file data into our buffer * @param fileName the file we open */ public Lexer(String fileName) { getInput(fileName); } public getNextToken(){ # no parameters } /*private functions for getIdentifier and getInteger. These helper functionsshould handle extracting the rest of an identifier/integer once one is identified. */ private getIdentifier(){ # no parameters } private getInteger(){ # no parameter } //A getAllTokens which calls getNextToken within a loop and returns a list of Tokens. public getAllTokens(){ # no parameters // // call getNextToken }
/** * Reads given file into the data member buffer * @param fileName name of file to parse */ private void getInput(String fileName) { try { Path filePath = Paths.get(fileName); byte[] allBytes = Files.readAllBytes(filePath); buffer = new String (allBytes); } catch (IOException e) { System.out.println ("You did not enter a valid file name in the run arguments."); System.out.println ("Please enter a string to be parsed:"); Scanner scanner = new Scanner(System.in); buffer=scanner.nextLine(); } } /** * Return all the token in the file * @return ArrayList of Token */ public ArrayList<Token> getAllTokens(){ //TODO: place your code here for lexing file return new ArrayList<Token>(); // don't forget to change the return statement } /** * Before your run this starter code * Select Run | Edit Configurations from the main menu. * In Program arguments add the name of file you want to test (e.g., test.txt) * @param args args[0] */ public static void main(String[] args) { String fileName=""; if (args.length==0) { System.out.println("You can test a different file by adding as an argument"); System.out.println("See comment above main"); System.out.println("For this run, test.txt used"); fileName="test.txt"; } else { fileName=args[0]; } Lexer lexer = new Lexer(fileName); // just print out the text from the file System.out.println(lexer.buffer); // here is where you'll call getAllTokens }}
public class Token {
public String type;
public String value;
public Token(String type, String value) {
this.type=type;
this.value=value;
}
public String toString(){
return type+" "+value;
}
}
HERE IS OTHER INFO ABOUT THIS ASSIGNMENT :A Lexical Analyzer (or Lexer for short) performs the task of finding the tokens in a large set of text.A token is a sequence of characters that form a single entity in a language. For an English sentence, each word is a token. For the SIMPLE coding language, there are a number of token types, including Identifier, AssignOp, Plus, and Integer. Your program will have a Token class that identifies two parts of a token, its type and its value.For instance, given the text “x23=32”, a Lexer would identify three tokens: an Identifier with value “x23”, an AssignOp with value “=” and an Integer with value “32”.Your Lexer will take a text file as input and create a list of the tokens contained in the file. The Lexer will identify the following token types:Identifier: starts with a letter followed by 0 or more letters and digits.Integer: a sequence of digitsAssignOp: a single ‘=’PlusOp: a single ‘+’UnknownOp: e.g., ‘$’EndOfFile: for every file, the last token generated by the Lexer will be this one.If your Lexer was provided the file:aa=34bb=45+6It will generate the following Tokens: Token Type Token ValueIdentifier aaAssignOp =Integer 34Identifier bbAssignOp =Integer 45PlusOp +Integer 6EndOfFile -
Unformatted Attachment Preview
Purchase answer to see full attachment
Explanation & Answer
Hello,Uploading the java files and te...