Skip to content

README.md

Latest commit

 

History

History
56 lines (48 loc) · 2.83 KB

README.md

File metadata and controls

56 lines (48 loc) · 2.83 KB

REPromptEngineering

Repository for all the code regarding the research about the influence of 'prompt smells' in prompt engineering.

Overview

Here are the most relevant files and paths:

  • /GPT-4 outputs/: All prompts and outputs we used in our study. For each research question and their scenarios, we clustered everything in the respective subfolders. Structure for the subfolders:
    • /csv/: All outputs from ChatGPT processed in a standardized CSV format
    • /output/: All outputs from ChatGPT, including all metadata
    • /prompt/: All prompts we used
    • /overview.csv: lists all prompt/output iterations with their respective UUID and the selected smells
  • /Groundtruth Code/src/groundtruth/DiceGame.java: Developed groundtruth code, based on our requirements.
  • /Groundtruth Tracings/: All expert tracings, including the unified and intersected versions.
  • /src/main/tracing.py: The code we used to do the automated prompting and analyzing (see Header "Features").
  • /Tracing Results/: All results for the tracing performance, including generated boxplots

Dependencies

we use the following python packages:

  • openai
  • python-dotenv
  • matplotlib
  • numpy

make sure to install them if you want to use our programm.

Features

Step 1: Trace Requirements

If you want to use our program for prompting with ChatGPT, you must do the following:

  • generate a "variables.env" file directly in the project folder and fill it with the following code: 
    API_KEY=[your_api_key]
ORG_KEY=[your_organization_key]
GSHEET_ID=[id_of_your_gsheet_with_all_requirements]
  • import the following packages:
    • google-api-python-client
    • google-auth-httplib2
    • google-auth-oauthlib
  • follow the steps from this gsheet python quickguide: https://developers.google.com/sheets/api/quickstart/python

In traying.py, you can adjust all relevant settings in the main function for Step 1:

  • tracing: Boolean, select if you want to do the tracing or skip it
  • smells: Smells (Enum), smells that you want to prompt
  • game: Game (Enum), the Study Object you want to use for the prompt
  • model: Model (Enum), select the ChatGPT model you want to use
  • temperature: int, temperature for the ChatGPT prompt. Between 0 and 1
  • top_p: top_p int, setting for ChatGPT. Between 0 and 1
  • iterations: int, number of iterations for each prompt

Step 2: Analyze the tracing & evaluate Performance

To analyze and evaluate all results from the /GPT-4 outputs/ folder. You can adjust the following settings:

  • analyzing: Boolean, select if you want to do the analysis or skip it
  • reference_tracings = list, select which References you want to consider the groundtruth (ReferenceTracing Enum)
  • smells: list of Smells (Enum), all smells that you want to analyze
  • create_boxplot = Boolean, choose if you want to create a boxplot