NEMDataTools
Overview
NEMDataTools Documentation
Overview
Installation
From PyPI (Recommended)
From TestPyPI (Pre-releases)
From Source (Development)
Requirements
Quick Start
Core Features
Development Status
✅
Completed Features
🚧
In Progress
📋
Tested Data Types
Documentation Structure
API Reference
Core Functions
License
NEM Data Profile
NEM Data Types
MMSDM URL Structure
Special Cases
AEMO Data Tables
Regional Reference Nodes
Data Access Frequency
Key Field Descriptions
Common Fields
Generation and Dispatch Fields
Bidding Fields
Data Model Relationships
Recent Changes
NEMDataTools Implementation Status
✅ Fully Supported Data Types
⚠️ Framework Ready (Implementation Complete, Testing Pending)
📋 Configured But Not Yet Tested
Data Quality Considerations
Developer Guide
NEMDataTools Project Board
Phase 1: Project Setup (Milestones 1-2) ✅
COMPLETED
Milestone 1: Development Environment Setup ✅
COMPLETED
Milestone 2: Core Module Skeletons ✅
COMPLETED
Phase 2: Core Functionality Implementation (Milestones 3-4)
Milestone 3: Time Utilities and Cache Management
Milestone 4: Data Downloading
Phase 3: Data Processing (Milestones 5-6)
Milestone 5: Basic Data Processing
Milestone 6: Advanced Processing
Phase 4: Documentation and Examples (Milestone 7)
Phase 5: Quality Assurance and Release (Milestone 8)
Phase 6: Continuous Improvement (Ongoing)
Development Practices
NEMDataTools Implementation Status
Core Components
1. Downloader Module
2. Cache Manager ✅
IMPLEMENTED
3. Time Utilities ✅
IMPLEMENTED
4. Data Processor
5. Data Type Handlers ✅
IMPLEMENTED
6. Batch Commands ✅
IMPLEMENTED
Implemented Functionalities:
Implementation Details:
Example Usage:
Key Parameters:
Error Handling Strategy:
Development Timeline
Implementation Details
AEMO Data Access
Caching Strategy
Error Handling
Future Extensions
Quick Start Guide: Setting Up NEMDataTools with UV
Prerequisites
Step 1: Install UV
Step 2: Clone and Setup the Project
Step 3: Create and Activate Virtual Environment
Step 4: Setup Configuration Files
1.
.uv.toml
2.
pyproject.toml
Step 5: Install Dependencies with UV
Step 6: Implement Core Modules
Step 7: Run Tests
Step 8: Code Formatting and Linting
Step 9: Build Documentation
Automation Script
Next Steps
Using UV for Dependency Management in NEMDataTools
Installing UV
Project Setup with UV
1. Create Project Structure
2. Create Virtual Environment with UV
3. Initialize Dependencies with UV
4. UV Configuration
Development Workflow with UV
Installing New Dependencies
Updating Dependencies
Managing Specific Versions
Locking Dependencies
Integration with CI/CD
Performance Considerations
Troubleshooting
Adding pre-commit to NEMDataTools
Step 1: Update
pyproject.toml
to include pre-commit
Step 2: Install pre-commit using UV
Step 3: Create a pre-commit configuration
Step 4: Install the pre-commit hooks
Step 5: Run pre-commit manually (optional)
Using pre-commit with UV
Integration with your development workflow
Customizing pre-commit configuration
Commit message convention with commitizen
Troubleshooting pre-commit
Additional Tips
Commitizen Conventional Commit Guide
What is Commitizen?
Commit Message Format
Types
Scope
Description
Examples
Creating Commits with Commitizen
Interactive Mode
Conventional git commit
Breaking Changes
Referencing Issues
Benefits of Using Commitizen
Commitizen CLI Commands
Integration with NEMDataTools
Customizing Commitizen
NEMDataTools
Index
Index