I suggest adding the topics cobol, llm, humaneval in the About section, as explained at https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/classifying-your-repository-with-topics

As mentioned in this blogpost, there is a FIM based way to prompt the models followed by rearranging the prompt + output so that the Working Storage Section and Linkage Section are in correct order. Do you plan to add support for this style of prompting in this repo?

I am assuming that the Pass@1 and %Compile scores mentioned in the blog are without using FIM based prompting?

Hi @ggordonhall

I can see in this blogpost that CodeLlama and mAInframer have been benchmarked on the COBOLEval benchmark. Is there any support in this repo to work directly with Huggingface model checkpoints ?

#2 finds different pass@1 results for mAInframer-7b compared to the one in the model card in https://huggingface.co/bloopai/mAInframer-7b (~4% vs original ~6%)

This PR fixes hf_complete to include the BOS token so the result can be reproduced

Add support for huggingface models.

Uses "cuda" as device. To test the 7b/13b/34b BloopAI models such as bloopai/mAInframer-7b, do:

    import data
    from generate import  HuggingfaceInfill, HuggingfaceComplete
    from utils import Model

    problems = data.read_problems()
    prompt = problems["HumanEval/1"]["prompt"]
    print(f"Prompt:\n\n\n{prompt}\n\n\n")

    model = Model(name="bloopai/mAInframer-7b", tokenizer="codellama/CodeLlama-7b-hf", prefix_token="<PRE>", suffix_token="<SUF>", middle_token="<MID>", eos_token="</s>")

    infiller = HuggingfaceInfill(model)
    completion = infiller.solve({"prompt": prompt})

    print(f"Completion:\n\n\n{completion}\n\n\n")

Prompt:

       IDENTIFICATION DIVISION.
       PROGRAM-ID. SEPARATE-PAREN-GROUPS.

       ENVIRONMENT DIVISION.
       
       INPUT-OUTPUT SECTION.

       DATA DIVISION.

       LINKAGE SECTION.

       01 LINKED-ITEMS.
           05 L-PAREN-STRING PIC X(100).
           05 RESULT OCCURS 100 TIMES INDEXED BY NI PIC X(100).

      * Input to this function is a string containing multiple groups of nested parentheses. Your goal is to
      * separate those group into separate strings and return the list of those.
      * Separate groups are balanced (each open brace is properly closed) and not nested within each other
      * Ignore any spaces in the input string.
      * >>> separate_paren_groups('( ) (( )) (( )( ))')
      * ['()', '(())', '(()())']
      * 

      * Complete the WORKING-STORAGE SECTION and the PROCEDURE DIVISION
      * Store the result in the RESULT variable and mark the end of your program with END PROGRAM

       WORKING-STORAGE SECTION.

Completion:

       IDENTIFICATION DIVISION.
       PROGRAM-ID. SEPARATE-PAREN-GROUPS.

       ENVIRONMENT DIVISION.
       
       INPUT-OUTPUT SECTION.

       DATA DIVISION.
       WORKING-STORAGE SECTION.

       01 WS-PAREN-STRING PIC X(100).
       01 WS-INDEX PIC 9(3) VALUE 1.
       01 WS-START-POSITION PIC 9(3) VALUE 1.
       01 WS-OPEN-COUNT PIC 9(3) VALUE 0.
       01 WS-RESULT PIC X(100).

       LINKAGE SECTION.

       01 LINKED-ITEMS.
           05 L-PAREN-STRING PIC X(100).
           05 RESULT OCCURS 100 TIMES INDEXED BY NI PIC X(100).

      * Input to this function is a string containing multiple groups of nested parentheses. Your goal is to
      * separate those group into separate strings and return the list of those.
      * Separate groups are balanced (each open brace is properly closed) and not nested within each other
      * Ignore any spaces in the input string.
      * >>> separate_paren_groups('( ) (( )) (( )( ))')
      * ['()', '(())', '(()())']
      * 

      * Store the result in the RESULT variable and mark the end of your program with END PROGRAM

       PROCEDURE DIVISION USING LINKED-ITEMS.
           MOVE L-PAREN-STRING TO WS-PAREN-STRING.
           PERFORM UNTIL WS-INDEX > LENGTH OF WS-PAREN-STRING
               IF WS-PAREN-STRING(WS-INDEX:1) = '(' THEN
                   ADD 1 TO WS-OPEN-COUNT
                   STRING WS-PAREN-STRING(WS-START-POSITION:WS-INDEX - WS-START-POSITION + 1) DELIMITED BY SIZE INTO WS-RESULT
                   MOVE WS-INDEX TO WS-START-POSITION
                   MOVE WS-RESULT TO RESULT(NI)
                   ADD 1 TO NI
               ELSE IF WS-PAREN-STRING(WS-INDEX:1) = ')' THEN
                   SUBTRACT 1 FROM WS-OPEN-COUNT
                   IF WS-OPEN-COUNT < 0 THEN
                       DISPLAY 'ERROR: Unbalanced Parentheses'
                       GOBACK
                   END-IF
               END-IF
               ADD 1 TO WS-INDEX
           END-PERFORM.
           GOBACK.

       END PROGRAM SEPARATE-PAREN-GROUPS.

And intermediate completion (before splitting PROCEDURE DIVISION and WORKING STORAGE):

       PROCEDURE DIVISION USING LINKED-ITEMS.
           MOVE L-PAREN-STRING TO WS-PAREN-STRING.
           PERFORM UNTIL WS-INDEX > LENGTH OF WS-PAREN-STRING
               IF WS-PAREN-STRING(WS-INDEX:1) = '(' THEN
                   ADD 1 TO WS-OPEN-COUNT
                   STRING WS-PAREN-STRING(WS-START-POSITION:WS-INDEX - WS-START-POSITION + 1) DELIMITED BY SIZE INTO WS-RESULT
                   MOVE WS-INDEX TO WS-START-POSITION
                   MOVE WS-RESULT TO RESULT(NI)
                   ADD 1 TO NI
               ELSE IF WS-PAREN-STRING(WS-INDEX:1) = ')' THEN
                   SUBTRACT 1 FROM WS-OPEN-COUNT
                   IF WS-OPEN-COUNT < 0 THEN
                       DISPLAY 'ERROR: Unbalanced Parentheses'
                       GOBACK
                   END-IF
               END-IF
               ADD 1 TO WS-INDEX
           END-PERFORM.
           GOBACK.

       END PROGRAM SEPARATE-PAREN-GROUPS.

<MID>       WORKING-STORAGE SECTION.

       01 WS-PAREN-STRING PIC X(100).
       01 WS-INDEX PIC 9(3) VALUE 1.
       01 WS-START-POSITION PIC 9(3) VALUE 1.
       01 WS-OPEN-COUNT PIC 9(3) VALUE 0.
       01 WS-RESULT PIC X(100).