# AgentNeo **Empower Your AI Applications with Unparalleled Observability and Optimization** AgentNeo is an advanced, open-source Python SDK for **Agentic AI Application Observability, Monitoring, and Evaluation Framework**. Designed to elevate your AI development experience, AgentNeo provides deep insights into your AI agents, Large Language Model (LLM) calls, and tool interactions. By leveraging AgentNeo, you can build more efficient, cost-effective, and high-quality AI-driven solutions.

Architecture of AgentNeo

### ⚡ Why AgentNeo? Whether you're a seasoned AI developer or just starting out, AgentNeo offers robust logging, visualization, and evaluation capabilities to help you debug and optimize your applications with ease. ### 🚀 Key Features * **Trace LLM Calls**: Monitor and analyze LLM calls from various providers like OpenAI and LiteLLM. * **Trace Agents and Tools**: Instrument and monitor your agents and tools to gain deeper insights into their behavior. * **Monitor Interactions**: Keep track of tool and agent interactions to understand system behavior. * **Detailed Metrics**: Collect comprehensive metrics on token usage, costs, and execution time. * **Flexible Data Storage**: Store trace data in SQLite databases and JSON log files for easy access and analysis. * **Simple Instrumentation**: Utilize easy-to-use decorators to instrument your code without hassle. * **Interactive Dashboard**: Visualize trace data and execution graphs in a user-friendly dashboard. * **Project Management**: Manage multiple projects seamlessly within the framework. * **Execution Graph Visualization**: Gain insights into your application's flow with detailed execution graphs. * **Evaluation Tools**: Assess and improve your AI agent's performance with built-in evaluation tools.

Dashboard

# Overview AgentNeo is an advanced, open-source Python SDK for Agentic AI Application Observability, Monitoring, and Evaluation Framework. It provides deep insights into your AI agents, Large Language Model (LLM) calls, and tool interactions, helping you build more efficient, cost-effective, and high-quality AI-driven solutions. ### Key Benefits * **Enhanced Visibility**: Get deep insights into your AI application's behavior * **Performance Optimization**: Identify and resolve bottlenecks * **Cost Control**: Monitor and optimize LLM usage and costs * **Quality Assurance**: Evaluate and improve agent performance * **Easy Integration**: Simple decorators for quick implementation ### Features at a Glance * **Tracing**: Monitor LLM calls, agent actions, and tool usage * **Interactive Dashboard**: Visualize execution flows and metrics * **Evaluation Framework**: Assess agent performance with built-in metrics * **Storage Solutions**: Flexible data storage with SQLite and JSON * **Project Management**: Organize and manage multiple projects # Quick Start Guide Get up and running with AgentNeo in minutes! ### Basic Setup ```python from agentneo import AgentNeo, Tracer, Evaluation, launch_dashboard # Create session and project neo_session = AgentNeo(session_name="my_session") neo_session.create_project(project_name="my_project") # Initialize tracer tracer = Tracer(session=neo_session) tracer.start() ``` ### Adding Traces #### Trace LLM Calls ```python @tracer.trace_llm("my_llm_call") async def get_llm_response(prompt): # Your LLM call here pass ``` #### Trace Tools ```python @tracer.trace_tool("my_tool") def calculate_metrics(data): # Your tool logic here pass ``` #### Trace Agents ```python @tracer.trace_agent("my_agent") def process_task(input_data): # Your agent logic here pass ``` ### Viewing Results ```python # Stop tracing tracer.stop() # Launch dashboard launch_dashboard(port=3000) ``` Visit `http://localhost:3000` to view your traces and metrics. ### Complete Example ```python from agentneo import AgentNeo, Tracer, Evaluation, launch_dashboard # Setup neo_session = AgentNeo(session_name="example_session") neo_session.create_project(project_name="example_project") tracer = Tracer(session=neo_session) tracer.start() # Define your functions @tracer.trace_tool("data_processor") def process_data(data): return {"processed": data} @tracer.trace_agent("main_agent") def main_agent(input_data): processed = process_data(input_data) return processed # Run your application result = main_agent({"data": "example"}) # Stop and view tracer.stop() launch_dashboard(port=3000) ``` # Installation Guide ### Prerequisites * Python 3.8 or above * pip, Conda, or Pipenv (latest version recommended) ### Quick Installation ```bash pip install agentneo ``` ### Platform-Specific Installation #### Windows ```bash # Create virtual environment python -m venv agentneo-env agentneo-env\Scripts\activate # Install AgentNeo pip install agentneo ``` #### macOS/Linux ```bash # Create virtual environment python -m venv agentneo-env source agentneo-env/bin/activate # Install AgentNeo pip install agentneo ``` #### Using Conda ```bash # Create conda environment conda create --name agentneo-env python=3.8 conda activate agentneo-env # Install AgentNeo pip install agentneo ``` ### Verification Verify your installation: ```python python -c "import agentneo; print(agentneo.__version__)" ``` ### Development Installation For contributing or development: ```bash git clone https://github.com/raga-ai-hub/agentneo.git cd agentneo pip install -e ".[dev]" ``` # Architecture ### System Overview AgentNeo follows a modular architecture designed for flexibility and extensibility:

AgentNeo's System Architecture

### Key Components #### 1. Tracer Module * Decorators for instrumentation * Event collection * Metric computation * Data buffering #### 2. Storage Layer * SQLite database * JSON log files * Data persistence * Query interface #### 3. Dashboard Service * Web interface * Real-time updates * Data visualization * Analysis tools #### 4. Evaluation * Metric computation * Performance analysis ### Data Flow 1. **Instrumentation** ```python @tracer.trace_llm() def your_function(): pass ``` 2. **Trace Collection** * Capture function calls * Record timestamps * Collect metrics 3. **Data Processing** * Compute metrics * Generate summaries * Update storage 4. **Visualization** * Display in dashboard # Basic Usage ### Setting Up AgentNeo ```python from agentneo import AgentNeo, Tracer # Initialize neo_session = AgentNeo(session_name="my_session") neo_session.create_project(project_name="my_project") # Start tracing tracer = Tracer(session=neo_session) tracer.start() ``` ### Using Decorators #### LLM Tracing ```python @tracer.trace_llm("text_generation") async def generate_text(prompt): response = await llm.generate(prompt) return response # Usage text = await generate_text("Tell me a story") ``` #### Tool Tracing ```python @tracer.trace_tool("data_processor") def process_data(data): # Processing logic return processed_data # Usage result = process_data(raw_data) ``` #### Agent Tracing ```python @tracer.trace_agent("task_agent") def execute_task(input_data): # Agent logic return result # Usage output = execute_task(data) ``` ### Dashboard Usage #### Launching ```python from agentneo import launch_dashboard # After your traces are complete tracer.stop() launch_dashboard(port=3000) ``` #### Viewing Data 1. Open `http://localhost:3000` 2. Navigate through different views: * Execution flow * Metrics * Timeline * Analytics # Components ### Tracer The core component for instrumenting and monitoring code. #### Features * Function decoration * Event tracking * Metric collection * Data buffering #### Usage ```python from agentneo import Tracer tracer = Tracer(session=neo_session) tracer.start() @tracer.trace_llm("my_llm") def llm_function(): pass @tracer.trace_tool("my_tool") def tool_function(): pass tracer.stop() ``` ### Dashboard Interactive web interface for visualization and analysis. #### Features * Real-time monitoring * Interactive graphs * Performance metrics * Detailed Trace views #### Access ```python from agentneo import launch_dashboard launch_dashboard(port=3000) ``` ### Storage Data persistence layer for traces and metrics. #### Features * SQLite database * JSON log files * Query interface * Data export #### Configuration ```python AgentNeo( session_name="my_session", storage_path="./data", storage_type="sqlite" ) ``` # Key Features ## Key Features ### Tracing Capabilities #### LLM Call Tracing Monitor and analyze LLM interactions with detailed metrics: * Input/output tokens * Response times * Cost tracking * Model parameters * Prompt analysis ```python @tracer.trace_llm("market_analysis") async def analyze_market(data): response = await openai.chat.completions.create( model="gpt-4o-mini", messages=[{"role": "user", "content": f"Analyze this market data: {data}"}] ) return response.choices[0].message.content ``` #### Tool Tracing Track tool usage and performance: * Execution time * Input/output validation * Error rates * Usage patterns ```python @tracer.trace_tool("data_processor") def process_market_data(raw_data): # Processing logic return processed_data ``` #### Agent Tracing Monitor agent behavior and decision-making: * Task decomposition * Tool selection * Goal achievement * Interaction patterns ```python @tracer.trace_agent("trading_agent") def execute_trade_strategy(market_conditions): # Trading logic return trade_decision ``` ### Monitoring Features #### Real-time Dashboard * Live execution tracking * Interactive visualizations * Performance metrics * Resource utilization #### Data Storage * SQLite backend * JSON log files * Custom storage adapters * Data export capabilities #### Analytics * Token usage trends * Cost analysis * Performance bottlenecks * Error patterns ### Evaluation Tools * Goal decomposition efficiency * Tool usage effectiveness * Response quality metrics * Cost optimization insights # Dashboard ## Overview Page

Screenshot of Overview Page

## Analytics Page

Screenshot of Analytics Page

## Trace History Page

Screenshot of Trace History Page

## Evaluation Page

Screenshot of Evaluation Page

# Advanced Usage ## Supported Metrics ### Core Metrics #### 1. Goal Decomposition Efficiency Measures how effectively an agent breaks down complex tasks. ```python exe.evaluate(metric_list=['goal_decomposition_efficiency']) ``` **Configuration Options:** ```python config = { "model": "gpt-4-turbo", "min_decomposition_steps": 2, "max_decomposition_steps": 10 } ``` #### 2. Goal Fulfillment Rate Assesses the success rate of achieving defined objectives. ```python exe.evaluate(metric_list=['goal_fulfillment_rate']) ``` **Configuration Options:** ```python config = { "success_threshold": 0.8, "partial_credit": True } ``` #### 3. Tool Call Correctness Rate Evaluates the accuracy of tool usage. ```python exe.evaluate(metric_list=['tool_call_correctness_rate']) ``` **Metadata Example:** ```python metadata = { "tools": [ { "name": "calculator", "expected_usage": ["multiplication", "division"] } ] } ``` #### 4. Tool Call Success Rate Measures the reliability of tool executions. ```python exe.evaluate(metric_list=['tool_call_success_rate']) ``` ### Using Multiple Metrics ```python # Evaluate multiple metrics together exe.evaluate( metric_list=[ 'goal_decomposition_efficiency', 'goal_fulfillment_rate', 'tool_call_correctness_rate', 'tool_call_success_rate' ] ) ``` # API Documentation This document provides an overview of the available API endpoints for retrieving project data, trace information, and system operations. It covers project data, trace information, and system operations. ### API Endpoints ### Base URL The base URL for all endpoints is: {{HOST\_URL}} ### API Overview > ### **GET** `/api/projects` This endpoint allows you to fetch all projects stored in the system. Each project contains metadata about the execution, including timing, cost, and token usage information. ### Details #### Response **Status Codes** * `200`: Success * `500`: Server error **200 Success** A list of projects with details like ID, project name, start and end times, duration, cost, and token usage. **Schema** ```json [ { "id": "integer", "project_name": "string", "start_time": "datetime", "end_time": "datetime", "duration": "float", "total_cost": "float", "total_tokens": "integer" } ] ``` **Example Response:** ```json [ { "id": 1, "project_name": "Document Analysis Project", "start_time": "2024-03-15T10:30:00", "end_time": "2024-03-15T10:35:00", "duration": 300.0, "total_cost": 0.15, "total_tokens": 1500 } ] ``` **500 Internal Server Error** A server error message if something goes wrong. **Schema** ```json { "error": "string" } ``` **Example Response:** ```json { "error": "Database connection error" } ``` ### Example Usage Python code to fetch and print all projects. #### Python ```python import requests def get_projects(): response = requests.get('{{HOST_URL}}/api/projects') if response.status_code == 200: return response.json() else: raise Exception(f"Error: {response.json()['error']}") # Usage try: projects = get_projects() for project in projects: print(f"Project: {project['project_name']}") print(f"Duration: {project['duration']} seconds") print(f"Cost: ${project['total_cost']}") print("---") except Exception as e: print(f"Failed to fetch projects: {str(e)}") ``` ## Get Project Details ### API Overview > ### **GET** `/api/projects/{project_id}` This endpoint allows you to fetch detailed information about a specific project, including all associated metadata and system information. ### Details #### Path Parameters | Parameter | Type | Required | Description | | ----------- | ------- | -------- | ------------------------------------ | | project\_id | integer | Yes | The unique identifier of the project | #### Response **Status Codes** * `200`: Success * `500`: Server error * `404`: Not Found **200 Success** Returns detailed project information, including system specs like OS, Python version, CPU, GPU, and installed packages. ```json { "id": "integer", "project_name": "string", "start_time": "datetime", "end_time": "datetime", "duration": "float", "total_cost": "float", "total_tokens": "integer", "system_info": { "os_name": "string", "os_version": "string", "python_version": "string", "cpu_info": "string", "gpu_info": "string", "disk_info": "string", "memory_total": "integer", "installed_packages": "string" } } ``` **Example Response:** ```json { "id": 1, "project_name": "Document Analysis Project", "start_time": "2024-03-15T10:30:00", "end_time": "2024-03-15T10:35:00", "duration": 300.0, "total_cost": 0.15, "total_tokens": 1500, "system_info": { "os_name": "Linux", "os_version": "Ubuntu 20.04", "python_version": "3.9.5", "cpu_info": "Intel(R) Core(TM) i7-10700K CPU @ 3.80GHz", "gpu_info": "NVIDIA GeForce RTX 3080", "disk_info": "500GB SSD", "memory_total": 32768, "installed_packages": "numpy==1.21.0,pandas==1.3.0" } } ``` **404 Not Found** Error message if the project is not found. **Schema** ```json { "error": "string" } ``` **Example Response:** ```json { "error": "Project not found" } ``` **500 Internal Server Error** Server error message. **Schema** ```json { "error": "string" } ``` **Example Response:** ```json { "error": "Database connection error" } ``` ### Example Usage Python code to fetch and display project details for a specific project ID. #### Python ```python import requests def get_project_details(project_id): response = requests.get(f'{{HOST_URL}}/api/projects/{project_id}') if response.status_code == 200: return response.json() elif response.status_code == 404: raise Exception("Project not found") else: raise Exception(f"Error: {response.json()['error']}") # Usage try: project_id = 1 project = get_project_details(project_id) print(f"Project: {project['project_name']}") print(f"Duration: {project['duration']} seconds") print(f"System: {project['system_info']['os_name']} {project['system_info']['os_version']}") print(f"Python Version: {project['system_info']['python_version']}") except Exception as e: print(f"Failed to fetch project details: {str(e)}") ``` ## Get Project Traces ### API Overview > ### **GET** `/api/projects/{project_id}/traces` This endpoint allows you to fetch all execution traces for a specific project, providing summary statistics and execution details like agent calls, LLM calls, tool calls, and errors for each trace. ### Details #### Path Parameters | Parameter | Type | Required | Description | | ----------- | ------- | -------- | ------------------------------------ | | project\_id | integer | Yes | The unique identifier of the project | #### Response #### Response **Status Codes** * `200`: Success * `500`: Server error **200 Success** A list of traces with details on execution times, agent/tool interactions, and errors. ```json [ { "id": "integer", "start_time": "datetime", "end_time": "datetime", "duration": "float", "total_agent_calls": "integer", "total_llm_calls": "integer", "total_tool_calls": "integer", "total_user_interactions": "integer", "total_errors": "integer" } ] ``` **Example Response:** ```json [ { "id": 1, "start_time": "2024-03-15T10:30:00", "end_time": "2024-03-15T10:31:00", "duration": 60.0, "total_agent_calls": 5, "total_llm_calls": 10, "total_tool_calls": 3, "total_user_interactions": 2, "total_errors": 0 } ] ``` **500 Internal Server Error** Server error message ```json { "error": "string" } ``` **Example Response:** ```json { "error": "Database connection error" } ``` ### Example Usage Python code to fetch and display project traces. #### Python ```python import requests def get_project_traces(project_id): response = requests.get(f'{{HOST_URL}}/api/projects/{project_id}/traces') if response.status_code == 200: return response.json() else: raise Exception(f"Error: {response.json()['error']}") # Usage try: project_id = 1 traces = get_project_traces(project_id) for trace in traces: print(f"Trace ID: {trace['id']}") print(f"Duration: {trace['duration']} seconds") print(f"Total LLM Calls: {trace['total_llm_calls']}") print(f"Total Tool Calls: {trace['total_tool_calls']}") print("---") except Exception as e: print(f"Failed to fetch project traces: {str(e)}") ``` ## Get Analysis Trace ### API Overview > ### **GET** `/api/analysis_traces/{trace_id}` This endpoint will fetch a detailed analysis of a specific trace, including calls to LLM models, tools, metrics, and system information. ### Details #### Path Parameters | Parameter | Type | Required | Description | | --------- | ------- | -------- | ---------------------------------- | | trace\_id | integer | Yes | The unique identifier of the trace | #### Response **Status Codes** * `200`: Success * `404`: Trace not found * `500`: Server error **200 Success** Includes LLM calls, tool calls, metrics, and system information related to the trace. ```json { "id": "integer", "project_id": "integer", "start_time": "datetime", "end_time": "datetime", "duration": "float", "llm_calls": [ { "id": "integer", "name": "string", "model": "string", "input_prompt": "string", "output": "string", "tool_call": "string", "start_time": "datetime", "end_time": "datetime", "duration": "float", "token_usage": "integer", "cost": "float", "memory_used": "float" } ], "tool_calls": [ { "id": "integer", "name": "string", "input_parameters": "object", "output": "string", "start_time": "datetime", "end_time": "datetime", "duration": "float", "memory_used": "float", "network_calls": "integer" } ], "metrics": [ { "id": "integer", "metric_name": "string", "score": "float", "reason": "string", "result_detail": "object", "config": "object", "start_time": "datetime", "end_time": "datetime", "duration": "float", "timestamp": "datetime" } ], "system_info": { "os_name": "string", "os_version": "string", "python_version": "string", "cpu_info": "string", "gpu_info": "string", "disk_info": "string", "memory_total": "float", "installed_packages": "object" } } ``` **Example Response:** ```json { "id": 123, "project_id": 2, "start_time": "2024-11-12T10:00:00Z", "end_time": "2024-11-12T10:30:00Z", "duration": 30.0, "llm_calls": [ { "id": 1, "name": "Text Analysis", "model": "gpt-4", "input_prompt": "Analyze sentiment: 'Stock prices are rising.'", "output": "Positive", "start_time": "2024-11-12T10:05:00Z", "end_time": "2024-11-12T10:05:10Z", "duration": 0.17, "token_usage": 100, "cost": 0.02, "memory_used": 128.0 } ], "tool_calls": [], "metrics": [], "system_info": {} } ``` **404 Not Found** Error message if the trace is not found. ```json { "error": "string" } ``` **Example Response:** ```json { "error": "Project not found" } ``` **500 Internal Server Error** Server error message. ```json { "error": "string" } ``` **Example Response:** ```json { "error": "Database connection error" } ``` #### Example Usage Python code snippet to initiate a new trace with the specified configurations. ```python import requests def get_analysis_trace(trace_id): response = requests.get(f'{{HOST_URL}}/api/analysis_traces/{trace_id}') if response.status_code == 200: return response.json() else: raise Exception(f"Error: {response.json()['error']}") # Usage try: trace_id = 123 trace = get_analysis_trace(trace_id) print(f"Trace ID: {trace['id']}") print(f"Project ID: {trace['project_id']}") print(f"Start Time: {trace['start_time']}") print(f"End Time: {trace['end_time']}") print(f"Duration: {trace['duration']} seconds") print(f"LLM Calls: {len(trace['llm_calls'])}") for llm in trace['llm_calls']: print(f" - LLM Call ID: {llm['id']}, Model: {llm['model']}, Duration: {llm['duration']} seconds") print("---") except Exception as e: print(f"Failed to fetch analysis trace: {str(e)}") ``` ## Get Trace Details ### API Overview > ### GET `/api/traces/{trace_id}` This endpoin returns comprehensive details about a specific trace, including all related calls and interactions. ### Details #### Path Parameters | Parameter | Type | Required | Description | | --------- | ------- | -------- | ---------------------------------- | | trace\_id | integer | Yes | The unique identifier of the trace | #### Response * `200`: Success * `404`: Trace not found * `500`: Server error **200 Success** **Schema** ```json { "id": "integer", "project_id": "integer", "start_time": "datetime", "end_time": "datetime", "duration": "float", "agent_calls": [ { "id": "integer", "name": "string", "start_time": "datetime", "end_time": "datetime", "llm_calls": ["array"], "tool_calls": ["array"], "user_interactions": ["array"], "errors": ["array"] } ], "system_info": { "os_name": "string", "os_version": "string", "python_version": "string", "cpu_info": "string", "gpu_info": "string", "disk_info": "string", "memory_total": "float", "installed_packages": "object" }, "tool_calls": [], "user_interaction": [], "error": [] "metrics": [], "system_info": {} } ``` **Example Response:** ```json { "id": 123, "project_id": 2, "start_time": "2024-11-12T10:00:00Z", "end_time": "2024-11-12T10:30:00Z", "duration": 30.0, "llm_calls": [ { "id": 1, "name": "Text Analysis", "model": "gpt-4", "input_prompt": "Analyze sentiment: 'Stock prices are rising.'", "output": "Positive", "start_time": "2024-11-12T10:05:00Z", "end_time": "2024-11-12T10:05:10Z", "duration": 0.17, "token_usage": 100, "cost": 0.02, "memory_used": 128.0 } ], "tool_calls": [], "metrics": [], "system_info": {} } ``` #### Example Usage: Python code to fetch and display all available traces, including their statuses and creation timestamps. ```python def get_trace_details(trace_id): response = requests.get(f'{{HOST_URL}}/api/traces/{trace_id}') if response.status_code == 200: return response.json() else: raise Exception(f"Error: {response.json()['error']}") # Usage try: trace_id = 123 trace = get_trace_details(trace_id) print(f"Trace ID: {trace['id']}, Duration: {trace['duration']}s") for agent in trace['agent_calls']: print(f"Agent Call ID: {agent['id']}, LLM Calls: {len(agent['llm_calls'])}") except Exception as e: print(f"Error: {e}") ``` ## Get Project Evaluation ### API Overview > ### **GET** `/api/projects/{project_id}/evaluation` This endpoint retrieves evaluation metrics for a specific project, with Trace filtering option. ### Details #### Path Parameters | Parameter | Type | Required | Description | | ----------- | ------- | -------- | ------------------------------------ | | project\_id | integer | Yes | The unique identifier of the project | #### Query Parameters | Parameter | Type | Required | Description | | --------- | ------ | -------- | ------------------------------------------------------------- | | trace\_id | string | No | Filter metrics by specific trace ID. Use 'all' for all traces | #### Response * `200`: Success * `500`: Server error **200 Success** **Schema** ```json [ { "trace_id": "integer", "metric_name": "string", "score": "float", "reason": "string", "result_detail": "object", "config": "object", "start_time": "datetime", "end_time": "datetime", "duration": "float", "timestamp": "datetime" } ] ``` **Example Response :** ```json [ { "trace_id": 1, "metric_name": "accuracy", "score": 0.95, "reason": "High accuracy achieved", "result_detail": {...}, "config": {...}, "start_time": "2024-01-01T00:00:00Z", "end_time": "2024-01-01T00:01:00Z", "duration": 60, "timestamp": "2024-01-01T00:01:00Z" } ] ``` **Example Usage** ```python import requests def get_project_evaluation(project_id, trace_id="all"): url = f'{{HOST_URL}}/api/projects/{project_id}/evaluation' params = {'trace_id': trace_id} response = requests.get(url, params=params) if response.status_code == 200: return response.json() else: raise Exception(f"Error {response.status_code}: {response.text}") try: project_id = 123 evaluation_data = get_project_evaluation(project_id, trace_id="all") print(evaluation_data) except Exception as e: print(f"API Error: {e}") ``` ## System Management ## Get Health Check ### API Overview > ### **GET** `/health` This is a simple endpoint used to check the health and availability of the server. It is typically used for monitoring and ensuring the API is responsive. ### Details **Response** * `"OK"`with a status code 200 to indicate that the server is healthy and operational. #### Example Usage To return the status of the port use this pyhton code ```python import requests def health_check(): url = '{{HOST_URL}}/health' response = requests.get(url) if response.status_code == 200: return response.text else: raise Exception(f"Error {response.status_code}: {response.text}") try: status = health_check() print(status) except Exception as e: print(f"API Error: {e}") ``` ### API Overview > ### **POST** `/api/cache/clear` ### Operation Overview This endpoint clears the server's cache storage. ### Operation Details #### Caching * Response is cached for 600 seconds (10 minutes) * Cache can be cleared using the /api/cache/clear endpoint #### Response * `200`: Success * `500`: Server error **200 Success** **Schema** ```json { "message": "string" } ``` **Example Response :** ```json { "message": "Cache cleared successfully" } ``` #### Example Usage: ```python import requests def clear_cache(): url = '{{HOST_URL}}/api/cache/clear' response = requests.post(url) if response.status_code == 200: return response.json() else: raise Exception(f"Error {response.status_code}: {response.text}") try: result = clear_cache() print(result['message']) except Exception as e: print(f"API Error: {e}") ``` ## Get Port ### API Overview > ### **GET** `/api/port` This endpoint returns the current port configuration of the server. It helps determine which port the server is running on, especially useful when there are multiple environments or when running on different servers. ### Details #### Response **200 Success** Returns the port number on which the server is currently running. ```json { "port": "string" } ``` **Example Response** ```json { "port": "3000" } ``` #### Example Usage ```python import requests def get_port(): url = '{{HOST_URL}}/api/port' response = requests.get(url) if response.status_code == 200: return response.json() else: raise Exception(f"Error {response.status_code}: {response.text}") try: port_info = get_port() print(f"Server is running on port: {port_info['port']}") except Exception as e: print(f"API Error: {e}") ``` ## Shutdown Server ### API Overview > ### **POST** `/api/shutdown` This endpoint allows for a graceful shutdown of the server. It ensures that all ongoing processes are completed before the server is stopped. ### Details #### Authentication * Only accessible from localhost (127.0.0.1, ::1, localhost) #### Response * `200`: Success * `403`: Forbidden (if not called from localhost) **200 Success** The server will begin the shutdown process. ```json { "message": "string" } ``` **Example Response** ```json { "message": "Server shutting down..." } ``` #### example Usage ```python import requests def shutdown_server(): url = '{{HOST_URL}}/api/shutdown' response = requests.post(url) if response.status_code == 200: return response.json() elif response.status_code == 403: raise Exception("Forbidden: Only accessible from localhost") else: raise Exception(f"Error {response.status_code}: {response.text}") try: shutdown_message = shutdown_server() print(shutdown_message['message']) except Exception as e: print(f"API Error: {e}") ``` **403 Forbidden** Returned when the request is not from localhost ### Example Usage #### Python ```python import requests class DashboardAPI: def __init__(self, base_url={{HOST_URL}}"): self.base_url = base_url def get_analysis_trace(self, trace_id): response = requests.get(f"{self.base_url}/api/analysis_traces/{trace_id}") if response.status_code == 200: return response.json() raise Exception(f"Error: {response.json().get('error', 'Unknown error')}") def get_trace(self, trace_id): response = requests.get(f"{self.base_url}/api/traces/{trace_id}") if response.status_code == 200: return response.json() raise Exception(f"Error: {response.json().get('error', 'Unknown error')}") def get_project_evaluation(self, project_id, trace_id=None): params = {"trace_id": trace_id} if trace_id else {} response = requests.get( f"{self.base_url}/api/projects/{project_id}/evaluation", params=params ) if response.status_code == 200: return response.json() raise Exception(f"Error: {response.json().get('error', 'Unknown error')}") def clear_cache(self): response = requests.post(f"{self.base_url}/api/cache/clear") if response.status_code == 200: return response.json() raise Exception(f"Error: {response.json().get('error', 'Unknown error')}") def get_port(self): response = requests.get(f"{self.base_url}/api/port") if response.status_code == 200: return response.json() raise Exception(f"Error: {response.json().get('error', 'Unknown error')}") # Usage Example try: api = DashboardAPI() # Get analysis trace trace_data = api.get_analysis_trace(1) print(f"Trace Analysis: {trace_data}") # Get project evaluation evaluation_data = api.get_project_evaluation(1, trace_id="all") print(f"Project Evaluation: {evaluation_data}") # Clear cache cache_result = api.clear_cache() print(f"Cache Clear Result: {cache_result}") except Exception as e: print(f"API Error: {str(e)}") ``` # Database Sheds light on the database schema ## Overview This database schema defines a set of models designed to capture various aspects of an application related to projects, user interactions, traces, system information, errors, and different types of calls (LLM, Tool, and Agent). These models have specific relationships and foreign key constraints to ensure data integrity and consistency. ## Database Models 1. UserInteractionModel 2. TraceModel 3. ProjectInfoModel 4. SystemInfoModel 5. ErrorModel 6. LLMCallModel 7. ToolCallModel 8. AgentCallModel 9. MetricModel The relationships between these models are established using SQLAlchemy's `relationship` function. ## Model Descriptions > ### `UserInteractionModel` This model represents a user's interaction with the application, such as an input or output. Each interaction is associated with a project, a trace, and agent. The model stores the following information: * `id`: Unique identifier for the user interaction. * `project_id`: Foreign key referencing the `ProjectInfoModel`. * `trace_id`: Foreign key referencing the `TraceModel`. * `agent_id`: Foreign key referencing the `AgentCallModel`. * `interaction_type`: The type of interaction, either 'input' or 'output'. * `content`: The content of the interaction. * `timestamp`: The timestamp of the interaction. **Relationships:** * `trace`: Relationship with `TraceModel`. * `project`: Relationship with `ProjectInfoModel`. * `agent`: Relationship with `AgentCallModel`. **Example usage:** ```python # Create a new user interaction user_interaction = UserInteractionModel( project_id=project.id, trace_id=trace.id, interaction_type='input', content='Hello, how are you?', timestamp=datetime.now() ) session.add(user_interaction) session.commit() ``` > ### `TraceModel` This model represents a trace of the application's execution. Each trace is associated with a project and can have various related entities, such as system information, errors, LLM calls, tool calls, agent calls, and user interactions. The model stores the following information: * `id`: Unique identifier for the trace. * `project_id`: Foreign key referencing the `ProjectInfoModel`. * `start_time`: The start time of the trace. * `end_time`: The end time of the trace. * `duration`: The duration of the trace. **Relationships:** `project`: Relationship with `ProjectInfoModel`. `system_info`: One-to-one relationship with `SystemInfoModel`. `errors`: One-to-many relationship with `ErrorModel`. `llm_calls`: One-to-many relationship with `LLMCallModel`. `tool_calls`: One-to-many relationship with `ToolCallModel`. `agent_calls`: One-to-many relationship with `AgentCallModel`. `user_interactions`: One-to-many relationship with `UserInteractionModel`. `metrics`: One-to-many relationship with `MetricModel`. **Example usage:** ```python # Create a new trace for the project trace = TraceModel(     project_id=project.id,     start_time=datetime.now(),     end_time=datetime.now() + timedelta(minutes=30),     duration=30.0 ) session.add(trace) session.commit() ``` > ### `ProjectInfoModel` This model represents information about a project, such as its name, start and end times, duration, total cost, and total tokens. The model has relationships with various other entities, such as traces, user interactions, LLM calls, tool calls, agent calls, and system information. The model stores the following information: * `id`: Unique identifier for the project. * `project_name`: The name of the project. * `start_time`: The start time of the project. * `end_time`: The end time of the project. * `duration`: The duration of the project. * `total_cost`: The total cost of the project. * `total_tokens`: The total number of tokens used in the project. **Relationships:** `traces`: One-to-many relationship with `TraceModel`. `llm_calls`: One-to-many relationship with `LLMCallModel`. `tool_calls`: One-to-many relationship with `ToolCallModel`. `agent_calls`: One-to-many relationship with `AgentCallModel`. `system_info`: One-to-one relationship with `SystemInfoModel`. `errors`: One-to-many relationship with `ErrorModel`. `user_interactions`: One-to-many relationship with `UserInteractionModel`. **Example usage:** ```python # Create a new project project = ProjectInfoModel(     project_name='My Project',     start_time=datetime.now(),     total_cost=1000.0,     total_tokens=10000 ) session.add(project) session.commit() ``` > ### `SystemInfoModel` This model represents information about the system on which the application was running, such as the operating system, Python version, CPU and GPU information, disk information, and installed packages. The model is associated with a project and a trace. The model stores the following information: * `id`: Unique identifier for the system information. * `project_id`: Foreign key referencing the `ProjectInfoModel`. * `os_name`: The name of the operating system. * `os_version`: The version of the operating system. * `python_version`: The version of Python. * `cpu_info`: Information about the CPU. * `gpu_info`: Information about the GPU (if available). * `disk_info`: Information about the disk. * `memory_total`: The total memory of the system. * `installed_packages`: A list of installed packages. * `trace_id`: Foreign key referencing the `TraceModel`. * **Relationships:** `trace`: Relationship with `TraceModel`. `project`: Relationship with `ProjectInfoModel`. #### Example usage: 
# Create a new system information entry
system_info = SystemInfoModel(
    project_id=project.id,
    os_name='Ubuntu',
    os_version='20.04',
    python_version='3.9.7',
    cpu_info='Intel Core i7-10700K',
    memory_total=32.0,
    installed_packages='numpy==1.21.5, pandas==1.3.4',
    trace_id=trace.id
)

session.add(system_info)

session.commit()
> ### `ErrorModel` This model represents errors that occurred during the application's execution. Each error is associated with a project, a trace, and possibly an agent, tool call, or LLM call. The model stores the following information: * `id`: Unique identifier for the error. * `project_id`: Foreign key referencing the `ProjectInfoModel`. * `trace_id`: Foreign key referencing the `TraceModel`. * `agent_id`: Foreign key referencing the `AgentCallModel`. * `tool_call_id`: Foreign key referencing the `ToolCallModel`. * `llm_call_id`: Foreign key referencing the `LLMCallModel`. * `error_type`: The type of error, either 'LLM', 'Tool', or 'Agent'. * `error_message`: The error message. * `timestamp`: The timestamp of the error. **Example usage:** ```python # Create a new error error = ErrorModel(     project_id=project.id,     trace_id=trace.id,     agent_id=agent.id,     error_type='LLM',     error_message='Invalid input format',     timestamp=datetime.now() ) session.add(error) session.commit() ``` > ### `LLMCallModel` This model represents a call to a Large Language Model (LLM). Each LLM call is associated with a project, an agent (if applicable), and a trace. The model stores the following information: * `id`: Unique identifier for the LLM call. * `project_id`: Foreign key referencing the `ProjectInfoModel`. * `agent_id`: Foreign key referencing the `AgentCallModel`. * `name`: The name of the LLM. * `model`: The model used for the LLM call. * `input_prompt`: The input prompt for the LLM call. * `output`: The output of the LLM call. * `tool_call`: The tool call associated with the LLM call (if applicable). * `start_time`: The start time of the LLM call. * `end_time`: The end time of the LLM call. * `duration`: The duration of the LLM call. * `token_usage`: The token usage for the LLM call. * `cost`: The cost of the LLM call. * `memory_used`: The memory used by the LLM call. * `trace_id`: Foreign key referencing the `TraceModel`. **Example usage:** ```python # Create a new LLM call llm_call = LLMCallModel(     project_id=project.id,     agent_id=agent.id,     name='GPT-4o-mini',     model='gpt-4o-mini',     input_prompt='What is the capital of France?',     output='The capital of France is Paris.',     start_time=datetime.now(),     end_time=datetime.now() + timedelta(seconds=5),     duration=5.0,     token_usage={'total': 20, 'prompt': 10, 'completion': 10},     cost={'total': 0.02, 'prompt': 0.01, 'completion': 0.01},     memory_used=512,     trace_id=trace.id ) session.add(llm_call) session.commit() ``` > ### `ToolCallModel` This model represents a call to a tool. Each tool call is associated with a project, an agent (if applicable), and a trace. The model stores the following information: * `id`: Unique identifier for the tool call. * `project_id`: Foreign key referencing the `ProjectInfoModel`. * `agent_id`: Foreign key referencing the `AgentCallModel`. * `name`: The name of the tool. * `input_parameters`: The input parameters for the tool call. * `output`: The output of the tool call. * `start_time`: The start time of the tool call. * `end_time`: The end time of the tool call. * `duration`: The duration of the tool call. * `memory_used`: The memory used by the tool call. * `trace_id`: Foreign key referencing the `TraceModel`. * `network_calls`: The network calls made by the tool (if applicable). **Example usage:** ```python # Create a new tool call tool_call = ToolCallModel(     project_id=project.id,     agent_id=agent.id,     name='Image Processor',     input_parameters='{"image_path": "test.jpg", "mode": "grayscale"}',     output='{"processed_image": "processed.jpg"}',     start_time=datetime.now(),     end_time=datetime.now() + timedelta(seconds=10),     duration=10.0,     memory_used=256,     trace_id=trace.id ) session.add(tool_call) session.commit() ``` > ### `AgentCallModel` This model represents a call to an agent. Each agent call is associated with a project and a trace, and can have related LLM calls, tool calls, and user interactions. The model stores the following information: * `id`: Unique identifier for the agent call. * `project_id`: Foreign key referencing the `ProjectInfoModel`. * `name`: The name of the agent. * `start_time`: The start time of the agent call. * `end_time`: The end time of the agent call. * `trace_id`: Foreign key referencing the `TraceModel`. * `llm_call_ids`: The IDs of the related LLM calls. * `tool_call_ids`: The IDs of the related tool calls. * `user_interaction_ids`: The IDs of the related user interactions. **Example usage:** ```python # Create a new agent call agent = AgentCallModel(     project_id=project.id,     name='Helpful Agent',     start_time=datetime.now(),     end_time=datetime.now() + timedelta(minutes=15),     trace_id=trace.id,     llm_call_ids=[llm_call.id],     tool_call_ids=[tool_call.id],     user_interaction_ids=[user_interaction.id] ) session.add(agent) session.commit() ``` > ### `MetricModel` This model represents a metric calculated for a trace. Each metric is associated with a trace and stores the following information: * `id`: Unique identifier for the metric. * `trace_id`: Foreign key referencing the `TraceModel`. * `metric_name`: The name of the metric. * `score`: The score of the metric. * `reason`: The reason for the metric score. * `result_detail`: Additional details about the metric result. * `config`: The configuration used to calculate the metric. * `start_time`: The start time of the metric calculation. * `end_time`: The end time of the metric calculation. * `duration`: The duration of the metric calculation. * `timestamp`: The timestamp of the metric calculation. **Example usage:** ```python # Create a new metric metric = MetricModel(     trace_id=trace.id,     metric_name='Response Quality',     score=4.5,     reason='Provided a clear and informative response',     result_detail={'sentiment': 'positive', 'coherence': 0.8},     config={'model': 'text-davinci-002', 'temperature': 0.7},     start_time=datetime.now(),     end_time=datetime.now() + timedelta(seconds=2),     duration=2.0,     timestamp=datetime.now() ) session.add(metric) session.commit() ``` ## Relationships Summary * **ProjectInfoModel** ↔ **TraceModel**: One-to-Many * **ProjectInfoModel** ↔ **UserInteractionModel**, **LLMCallModel**, **ToolCallModel**, **AgentCallModel**: One-to-Many * **ProjectInfoModel** ↔ **SystemInfoModel**: One-to-One * **TraceModel** ↔ **UserInteractionModel**, **ErrorModel**, **LLMCallModel**, **ToolCallModel**, **AgentCallModel**: One-to-Many * **AgentCallModel** ↔ **LLMCallModel**, **ToolCallModel**, **UserInteractionModel**: One-to-Many # Metric Configuration ## Metric Configuration ### Basic Configuration ```python # Standard configuration config = { "model": "gpt-4-turbo", "threshold": 0.8, "detailed_output": True } # Standard metadata metadata = { "tools": [ { "name": "tool_name", "description": "tool_description" } ] } # Evaluate with config exe.evaluate( metric_list=['metric_name'], config=config, metadata=metadata ) ``` ### Advanced Configuration Options #### Global Settings ```python global_config = { "model": "gpt-4-turbo", "temperature": 0.7, "evaluation_mode": "strict", "logging_level": "detailed" } ``` #### Metric-Specific Settings ```python metric_config = { "goal_decomposition_efficiency": { "min_steps": 2, "max_steps": 10, "complexity_threshold": 0.7 }, "tool_call_correctness_rate": { "strict_mode": True, "partial_credit": False } } ``` #### Custom Metadata ```python custom_metadata = { "application_context": "financial_analysis", "expected_behaviors": ["data_analysis", "prediction"], "tools": [ { "name": "market_analyzer", "expected_usage": ["analysis", "prediction"], "critical_operations": ["risk_assessment"] } ] } ``` # Overview ### Introduction AgentNeo provides comprehensive metrics and evaluation capabilities to assess and optimize your AI applications' performance. The evaluation framework helps you understand your agents' behavior, efficiency, and effectiveness. ### Key Capabilities * Performance Assessment * Cost Analysis * Quality Metrics * Behavioral Analysis * Custom Metric Support ### Getting Started with Metrics ```python from agentneo import Evaluation # Initialize evaluation exe = Evaluation(session=neo_session, trace_id=tracer.trace_id) # Run evaluation exe.evaluate(metric_list=['goal_decomposition_efficiency']) # Get results results = exe.get_results() ``` ### Evaluation Process 1. Collect trace data 2. Compute metrics 3. Generate insights 4. Provide recommendations # Supported Metrics ## Supported Metrics ### Core Metrics #### 1. Goal Decomposition Efficiency Measures how effectively an agent breaks down complex tasks. ```python exe.evaluate(metric_list=['goal_decomposition_efficiency']) ``` **Configuration Options:** ```python config = { "model": "gpt-4o-mini", } ``` #### 2. Goal Fulfillment Rate Assesses the success rate of achieving defined objectives. ```python exe.evaluate(metric_list=['goal_fulfillment_rate']) ``` **Configuration Options:** ```python config = { "model": "gpt-4o-mini", } ``` #### 3. Tool Call Correctness Rate Evaluates the accuracy of tool usage. ```python exe.evaluate(metric_list=['tool_call_correctness_rate']) ``` **Metadata Example:** ```python metadata = { "tools": [ { "name": "calculator", "expected_usage": ["multiplication", "division"] } ] } ``` #### 4. Tool Call Success Rate Measures the reliability of tool executions. ```python exe.evaluate(metric_list=['tool_call_success_rate']) ``` ### Using Multiple Metrics ```python # Evaluate multiple metrics together exe.evaluate( metric_list=[ 'goal_decomposition_efficiency', 'goal_fulfillment_rate', 'tool_call_correctness_rate', 'tool_call_success_rate' ] ) ``` # Guidelines ### Code Standards #### Python Style Guide * Follow PEP 8 * Use type hints * Maximum line length: 88 characters (Black default) * Use docstrings for all public functions/classes #### Example ```python from typing import Dict, Optional def process_data( data: Dict[str, any], config: Optional[Dict[str, any]] = None ) -> Dict[str, any]: """ Process input data according to configuration. Args: data: Input data dictionary config: Optional configuration parameters Returns: Processed data dictionary """ # Implementation pass ``` ### Documentation * Update relevant documentation * Add docstrings to new code * Include examples for new features * Update changelog ### Testing * Write unit tests for new features * Maintain or improve coverage * Include integration tests when needed ### Git Workflow 1. Fork the repository 2. Create feature branch 3. Make changes 4. Run tests 5. Submit pull request ### Pull Request Process 1. Update documentation 2. Add tests 3. Run linting and formatting 4. Update changelog 5. Request review # Pull Request Process ### Before Creating a PR 1. **Update Your Fork** ```bash git remote add upstream https://github.com/raga-ai-hub/agentneo.git git fetch upstream git rebase upstream/main ``` 2. **Run Tests** ```bash pytest tests/ pytest --cov=agentneo tests/ ``` 3. **Check Code Style** ```bash black . isort . flake8 ``` ### Creating a Pull Request #### PR Checklist * [ ] Updated documentation * [ ] Added/updated tests * [ ] Ran code formatting tools * [ ] Updated changelog * [ ] Resolved merge conflicts * [ ] Added type hints * [ ] Included docstrings #### PR Template ```markdown ## Description Brief description of changes ## Type of Change - [ ] Bug fix - [ ] New feature - [ ] Documentation update - [ ] Performance improvement - [ ] Code cleanup ## Test Plan Describe how you tested your changes ## Breaking Changes List any breaking changes ## Additional Notes Any extra information ``` ### Review Process #### What We Look For 1. Code Quality * Clean and readable code * Proper error handling * Efficient implementations 2. Testing * Adequate test coverage * Edge case handling * Performance considerations 3. Documentation * Clear docstrings * Updated README/docs * Code comments where needed #### Review Timeline * Initial review: 1-2 business days * Follow-up reviews: 1 business day * Final approval: 1-2 business days ### After Approval 1. **Squash and Merge** ```bash git rebase -i main git push -f origin feature-branch ``` 2. **Update Local Repository** ```bash git checkout main git pull upstream main ``` 3. **Clean Up** ```bash git branch -d feature-branch git push origin --delete feature-branch ``` # Development Setup ### Initial Setup 1. Clone the Repository ```bash git clone https://github.com/raga-ai-hub/agentneo.git cd agentneo ``` 2. Create Virtual Environment ```bash python -m venv venv source venv/bin/activate # Unix # or venv\Scripts\activate # Windows ``` 3. Install Development Dependencies ```bash pip install -e ".[dev]" ``` ### Development Tools #### Required Tools * Python 3.8+ * Git * pytest * black (code formatting) * isort (import sorting) * flake8 (linting) #### Installing Development Tools ```bash pip install black isort flake8 pytest pytest-cov ``` ### Setting Up Pre-commit Hooks ```bash pre-commit install ``` ### Running Tests ```bash # Run all tests pytest # Run with coverage pytest --cov=agentneo tests/ # Run specific test file pytest tests/test_tracer.py ``` ### To run the dashboard after local UI code changes ```bash # 1. Move to the UI folder > cd agentneo/ui # 2. Install the dependencies > npm i # 3. Build the UI files after the code changes > npm run build # 4. Go back to the root folder > cd .. # 5. Remove any previous installations of agentneo > pip uninstall agentneo # 6. Install the updated code > pip install -e . ``` # Common Issues and Solutions ### Installation Issues #### Problem: Import Errors ```python ImportError: No module named 'agentneo' ``` **Solution:** 1. Verify installation: ```bash pip list | grep agentneo ``` 2. Reinstall package: ```bash pip uninstall agentneo pip install agentneo ``` #### Problem: Version Conflicts **Solution:** ```bash pip install --upgrade agentneo pip install --upgrade "agentneo[all]" ``` ### Tracing Issues #### Problem: Decorators Not Working ```python @tracer.trace_llm() def my_function(): # Function not being traced ``` **Solution:** 1. Verify tracer initialization ```python # Correct initialization neo_session = AgentNeo(session_name="my_session") neo_session.create_project("my_project") tracer = Tracer(session=neo_session) tracer.start() # Don't forget this! ``` #### Problem: Missing Data in Dashboard **Solution:** 1. Check tracer status 2. Verify data storage 3. Ensure proper stopping ```python # Correct order tracer.stop() # Stop before launching dashboard launch_dashboard(port=3000) ``` ### Dashboard Issues #### Problem: Connection Refused **Solution:** 1. Check port availability 2. Verify server status 3. Try alternative port ```python launch_dashboard(port=3001) # Try different port ``` ### Evaluation Issues #### Problem: Metrics Not Computing **Solution:** 1. Verify trace data exists 2. Check metric configuration ```python # Correct usage exe = Evaluation( session=neo_session, trace_id=tracer.trace_id # Verify trace_id exists ) ``` # Frequently Asked Questions ### General Questions #### Q: What Python versions are supported? A: AgentNeo supports Python 3.8 and above. #### Q: Can I use AgentNeo with any LLM provider? A: Yes, AgentNeo supports various providers including OpenAI, Anthropic, and others through LiteLLM. #### Q: How do I update AgentNeo? A: Use pip to update to the latest version: ```bash pip install --upgrade agentneo ``` ### Privacy & Security #### Q: Is my data safe with AgentNeo? What about privacy? A: Yes, your data is completely safe. AgentNeo operates entirely on your local system and does not transmit any data externally. All data, including traces and logs, are stored locally on your machine. We do not collect any telemetry data or usage statistics - your privacy is our priority. The only external communications are the API calls you explicitly configure to your chosen LLM provider (like OpenAI or Anthropic). These calls are made using your own API keys and follow the privacy policies of those providers. #### Q: Where is my data stored? A: All data is stored locally on your system: * Trace data: `{agentneo-package-path}/ui/dist/trace_data.db` * Logs: In your system's standard logging directory * Configuration: In your project directory You have complete control over this data and can delete it at any time. // ... existing questions ... ### Privacy & Security #### Q: Is my data safe with AgentNeo? What about privacy? A: Yes, your data is completely safe. AgentNeo operates entirely on your local system and does not transmit any data externally. All data, including traces and logs, are stored locally on your machine. We do not collect any telemetry data or usage statistics - your privacy is our priority. The only external communications are the API calls you explicitly configure to your chosen LLM provider (like OpenAI or Anthropic). These calls are made using your own API keys and follow the privacy policies of those providers. #### Q: Where is my data stored? A: All data is stored locally on your system: * Trace data: `{agentneo-package-path}/ui/dist/trace_data.db` * Logs: In your system's standard logging directory * Configuration: In your project directory You have complete control over this data and can delete it at any time. ### Integration Questions #### Q: Can I use AgentNeo with other Agentic Framworks? A: Yes, see the Developer Guide for more information. ### Customization Questions #### Q: How do I open the dashboard on a different port? A: You can specify a custom port when launching the dashboard: ```python neo_session.launch_dashboard(port=8080) ``` # Breaking Changes ### Version 1.2.0 #### Evaluation Framework * Renamed `Execution` class to `Evaluation` * Updated metric computation methods ```python # Old code (1.1.2) from agentneo import Execution exe = Execution(session=neo_session) # New code (1.2.0) from agentneo import Evaluation eval = Evaluation(session=neo_session) ``` #### Configuration Changes * Modified dashboard configuration structure * Updated storage backend access method. Now the the Database is access through API Endpoints. This change is currently implemented in the Analytics and Trace History Pages. Gradually the access of DB from frontend TypeScript(.ts) would be completely phased out. ### Version 1.1.0 #### API Changes * Restructured Tracing * Modified event handling system * Remove node.js, npm dependency for dashboard ```python # Old code (1.0.0) tracer = Tracer() tracer.init(session) # New code (1.1.0) tracer = Tracer(session=session) ``` #### Storage Updates * New SQLite schema * Changed JSON log format ### Migration Guidelines See Migration Guide for detailed upgrade instructions. # Version History ### Latest Release #### 1.2.0 (Current) **Release Date:** October 28, 2024 **Major Features** * Add Flask Server to server API Endpoints for Database access, removing the dependence on frontend to * Add Analytics Page to view Trace Analytics using new API endpoints for database access. * Rewrote the Trace History page to use APIs for database access * Add Trace Details Panel to `Trace History` and `Evaluation` pages * Improved Evaluation Metrics * Enhanced various dashboard components * Improved error handling * Add more Documentation * Add integration examples for LangGraph, OpenAI Swarm **Minor Improvements** * Performance optimizations * Better error messages * Documentation updates * Bug fixes ### Previous Releases #### 1.1.2 **Release Date:** October 21, 2024 **Changes** * Moved travel\_planner.ipynb to examples/travel\_planner.ipynb * Various bug fixes and improvements #### 1.1.1 **Release Date:** October 17, 2024 **Major Changes** * Added evaluation metrics support * Enhanced UI improvements * Added new examples (CrewAI, Autogen, Digital Marketing) * Improved tracing capabilities **Minor Improvements** * Added GPU and disk info collection * Enhanced output formatting * Added tool\_call and function\_call info in trace\_llm output * Fixed tracer stop error in FinancialAnalysisSystem.ipynb * Removed npm and nodejs dependencies * Added wrapt dependency #### 1.1.0 **Release Date:** October 17, 2024 **Features** * Interactive dashboard * SQLite storage backend * Enhanced tracing capabilities * Tool usage analytics **Bug Fixes** * Fixed memory leak in long-running traces * Corrected timestamp handling * Resolved concurrent tracing issues #### 1.0.0 **Release Date:** September 28, 2024 **Initial Features** * Basic tracing functionality * LLM call monitoring * Tool and agent tracing * Simple metrics collection # Glossary Core Concepts #### A * **Agent**: An autonomous component that performs specific tasks * **AgentNeo**: The main framework for AI application observability #### D * **Dashboard**: Web interface for visualizing trace data and metrics * **Decorator**: Python syntax for adding tracing functionality #### E * **Evaluation**: Framework for assessing agent performance * **Event**: A tracked occurrence in the system #### L * **LLM**: Large Language Model * **Log**: Record of system events and traces #### M * **Metric**: Measurement of system or agent performance * **Monitoring**: Real-time observation of system behavior #### P * **Project**: Logical grouping of related traces * **Provider**: LLM service provider (e.g., OpenAI) #### S * **Session**: Container for related traces and projects * **Storage**: System for persisting trace data #### T * **Trace**: Record of execution flow and performance metrics * **Tool**: Function or service that agents can use ### Technical Terms #### Performance Metrics * **Latency**: Time taken for operation completion * **Throughput**: Rate of operation processing * **Token Usage**: Number of tokens consumed in LLM calls #### System Components * **Buffer**: Temporary storage for trace data * **Flush**: Process of writing buffered data to storage * **Hook**: Extension point for custom functionality # Changelog ## v1.2.x ### Major Features * Add Flask Server to server API Endpoints for Database access, removing the dependence on frontend to * Add Analytics Page to view Trace Analytics using new API endpoints for database access. * Rewrote the Trace History page to use APIs for database access * Add Trace Details Panel to `Trace History` and `Evaluation` pages * Improved Evaluation Metrics * Enhanced various dashboard components * Improved error handling * Add more Documentation * Add integration examples for LangGraph, OpenAI Swarm ### Minor Improvements * Performance optimizations * Better error messages * Documentation updates * Bug fixes