# Awesome Tool

Here’s a beginner-friendly **Awesome List** on creating, designing, and maintaining tools for bug bounty and cybersecurity development. This list focuses on a streamlined approach with templates, checklists, and best practices.

***

## **Awesome Tool Creation for Cybersecurity**

> A curated list of resources, templates, and checklists to help you quickly create tools that are functional, good-looking, and easy to maintain.

***

### **Table of Contents**

1. [Introduction](#introduction)
2. [Development Basics](#development-basics)
3. [Pre-Built Templates](#pre-built-templates)
4. [Tool Design Checklist](#tool-design-checklist)
5. [Automation and Recon Scripting](#automation-and-recon-scripting)
6. [User Interface Tips](#user-interface-tips)
7. [DevOps and CI/CD Integration](#devops-and-cicd-integration)
8. [Resources for Inspiration](#resources-for-inspiration)

***

### **1. Introduction**

Building custom tools for bug bounty or cybersecurity tasks is a skill that can save time, reduce errors, and improve efficiency. This list will guide you through creating tools with:

* Proper documentation.
* Easy-to-use interfaces.
* Scalability for future updates.

***

### **2. Development Basics**

#### **Languages to Learn**

1. **Python**: Great for automation and scripting.
2. **Bash**: Perfect for lightweight scripts and command-line tools.
3. **Go**: Efficient and fast for building scalable tools.

#### **Environment Setup**

1. Install essential software:

   ```bash
   sudo apt update && sudo apt install -y git curl python3 python3-pip golang
   ```
2. Set up a project structure:

   ```bash
   mkdir -p ~/tools/mytool/{src,docs,tests}
   ```

#### **Version Control**

1. Initialize Git:

   ```bash
   git init
   ```
2. Use branches for new features:

   ```bash
   git checkout -b feature/my-feature
   ```

***

### **3. Pre-Built Templates**

#### **CLI Tool Template**

```bash
#!/bin/bash
# MyTool: A simple CLI example

target=$1
output_dir="./output"

# Ensure target is specified
if [ -z "$target" ]; then
  echo "Usage: ./mytool.sh <target>"
  exit 1
fi

# Create output directory
mkdir -p $output_dir

# Example task
echo "[+] Scanning $target"
nmap -sV -oN $output_dir/nmap_$target.txt $target
```

#### **Python Recon Script**

```python
import subprocess
import sys

def check_tool(tool_name):
    result = subprocess.run(["which", tool_name], capture_output=True, text=True)
    if not result.stdout.strip():
        print(f"Error: {tool_name} not found. Install it first!")
        sys.exit(1)

def main(target):
    output_file = f"./output/{target}_scan.txt"
    print(f"[+] Scanning {target}")
    subprocess.run(["nmap", "-sV", "-oN", output_file, target])

if __name__ == "__main__":
    if len(sys.argv) != 2:
        print("Usage: python recon.py <target>")
        sys.exit(1)

    check_tool("nmap")
    main(sys.argv[1])
```

***

### **4. Tool Design Checklist**

#### **Basic Features**

1. Accept input via command-line arguments.
2. Include usage instructions (`--help` flag).
3. Validate user input (e.g., ensure the domain is valid).

#### **Structure**

* **src/**: Codebase.
* **docs/**: Documentation files (README, usage guides).
* **tests/**: Unit and functional tests.

#### **Error Handling**

* Log errors to a file:

  ```bash
  command_here 2>> error.log
  ```
* Exit on failure with meaningful messages.

***

### **5. Automation and Recon Scripting**

#### **Recon Pipeline Example**

```bash
#!/bin/bash

target=$1
output_dir="./recon/$target"

if [ -z "$target" ]; then
  echo "Usage: $0 <target>"
  exit 1
fi

mkdir -p $output_dir

# Subdomain enumeration
echo "[+] Enumerating subdomains..."
subfinder -d $target -silent > $output_dir/subdomains.txt

# Probing live domains
echo "[+] Probing live domains..."
cat $output_dir/subdomains.txt | httpx -silent > $output_dir/live_domains.txt

# Scanning for vulnerabilities
echo "[+] Running Nuclei..."
nuclei -l $output_dir/live_domains.txt -t ~/nuclei-templates -o $output_dir/nuclei_results.txt
```

***

### **6. User Interface Tips**

#### **Command-Line Features**

1. Add colorful output:

   ```bash
   echo -e "\033[0;32m[+] Task Completed\033[0m"
   ```
2. Include a progress bar for long tasks.

#### **Customizing Output**

1. Use tables for clarity:

   ```bash
   printf "%-15s %-10s\n" "Domain" "Status"
   printf "%-15s %-10s\n" "example.com" "Live"
   ```
2. Generate Markdown reports:

   ```bash
   echo "# Report for $target" > report.md
   echo "- Live Domains: $(wc -l live_domains.txt)" >> report.md
   ```

***

### **7. DevOps and CI/CD Integration**

#### **GitHub Actions Pipeline**

```yaml
name: Tool Deployment

on:
  push:
    branches:
      - main

jobs:
  build-and-deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Code
        uses: actions/checkout@v3

      - name: Install Dependencies
        run: |
          sudo apt update
          sudo apt install -y nmap subfinder nuclei

      - name: Run Tool
        run: ./mytool.sh example.com
```

***

### **8. Resources for Inspiration**

#### **Tool Repositories**

* [Subfinder](https://github.com/projectdiscovery/subfinder)
* [Nuclei](https://github.com/projectdiscovery/nuclei)
* [FFuF](https://github.com/ffuf/ffuf)

#### **Cheatsheets**

* [HackTricks](https://book.hacktricks.xyz/)
* [PayloadAllTheThings](https://github.com/swisskyrepo/PayloadsAllTheThings)

#### **Books**

* "Automate the Boring Stuff with Python" by Al Sweigart.
* "Black Hat Python" by Justin Seitz.

***

This list is a starting point for building your tools and automating tasks. Feel free to customize and expand it for your needs. 🚀

***

> Comprehensive guidelines for building effective, scalable, and user-friendly tools, focusing on bug bounty, cybersecurity, and automation.

***

### **Table of Contents**

1. [Introduction](#introduction)
2. [Getting Started with Tool Development](#getting-started-with-tool-development)
3. [Tool Design Fundamentals](#tool-design-fundamentals)
4. [Advanced Automation Scripts](#advanced-automation-scripts)
5. [Scripting Best Practices](#scripting-best-practices)
6. [Integrating APIs in Tools](#integrating-apis-in-tools)
7. [Report Generation](#report-generation)
8. [Testing and Debugging](#testing-and-debugging)
9. [Useful Libraries and Frameworks](#useful-libraries-and-frameworks)
10. [Resources and Learning Platforms](#resources-and-learning-platforms)

***

### **1. Introduction**

Building tools isn't just about automating tasks; it's about creating reliable, efficient, and reusable workflows. This list provides detailed steps to go from zero to building highly functional tools for bug bounty and security research.

***

### **2. Getting Started with Tool Development**

#### **Languages to Learn**

* **Python**: Ideal for rapid development and API integration.
* **Bash**: Great for quick and lightweight automation scripts.
* **Go (Golang)**: Perfect for building high-performance tools.
* **JavaScript**: Use for browser automation and interacting with web apps.

#### **Tools to Install**

1. **Git** for version control:

   ```bash
   sudo apt install git
   ```
2. **Package managers**:
   * Python: `pip` and `pipenv`
   * JavaScript: `npm` or `yarn`
   * Go: `go install`
3. **Development Environment**:
   * IDE: VSCode, PyCharm, or IntelliJ.
   * Linters: `flake8` for Python, `shellcheck` for Bash, and `golangci-lint` for Go.

***

### **3. Tool Design Fundamentals**

#### **Key Design Principles**

1. **User-friendly interface**:
   * Use clear command-line options (`-h`, `--help`).
   * Add error messages for incorrect inputs.
2. **Scalable architecture**:
   * Use modular functions.
   * Store reusable logic in libraries.
3. **Comprehensive output**:
   * Include color-coded CLI output.
   * Generate reports in multiple formats (JSON, CSV, Markdown).

#### **Directory Structure for Projects**

```plaintext
mytool/
├── README.md          # Documentation
├── LICENSE            # License file
├── requirements.txt   # Python dependencies
├── main.py            # Entry point
├── src/               # Core logic
│   ├── utils.py       # Helper functions
│   ├── modules/       # Submodules
├── tests/             # Unit tests
├── config/            # Configuration files
└── output/            # Scan results
```

***

### **4. Advanced Automation Scripts**

#### **Full Recon Workflow**

```bash
#!/bin/bash

domain=$1
output_dir="./output/$domain"

if [ -z "$domain" ]; then
  echo "Usage: $0 <domain>"
  exit 1
fi

mkdir -p $output_dir

echo "[+] Enumerating subdomains..."
subfinder -d $domain -silent > $output_dir/subdomains.txt
amass enum -passive -d $domain >> $output_dir/subdomains.txt

echo "[+] Checking live hosts..."
cat $output_dir/subdomains.txt | httpx -silent > $output_dir/live_hosts.txt

echo "[+] Running Nuclei..."
nuclei -l $output_dir/live_hosts.txt -t ~/nuclei-templates -o $output_dir/nuclei_results.txt
```

#### **Scheduled Recon with Crontab**

1. Edit the crontab:

   ```bash
   crontab -e
   ```
2. Add a daily schedule:

   ```bash
   0 2 * * * /path/to/your/script.sh >> /path/to/logfile.log 2>&1
   ```

***

### **5. Scripting Best Practices**

#### **Bash**

* **Set strict mode** to catch errors:

  ```bash
  set -euo pipefail
  ```
* **Handle arguments**:

  ```bash
  while getopts "d:o:" opt; do
    case $opt in
      d) domain=$OPTARG ;;
      o) output_dir=$OPTARG ;;
      *) echo "Invalid option"; exit 1 ;;
    esac
  done
  ```

#### **Python**

* Use `argparse` for CLI tools:

  ```python
  import argparse

  parser = argparse.ArgumentParser(description="Tool Description")
  parser.add_argument("-d", "--domain", help="Target domain", required=True)
  args = parser.parse_args()
  ```
* Leverage virtual environments:

  ```bash
  python3 -m venv venv
  source venv/bin/activate
  ```

***

### **6. Integrating APIs in Tools**

#### **API Keys**

* Store them in environment variables:

  ```bash
  export API_KEY="your_api_key"
  ```
* Access in Python:

  ```python
  import os
  api_key = os.getenv("API_KEY")
  ```

#### **Example API Call**

Using Python's `requests` library:

```python
import requests

url = "https://api.shodan.io/shodan/host/search"
params = {"key": "your_api_key", "query": "apache"}
response = requests.get(url, params=params)

print(response.json())
```

***

### **7. Report Generation**

#### **Markdown Report Template**

```markdown
# Recon Report: [Target]

## Summary
- Total Subdomains Found: XX
- Live Hosts: XX
- Vulnerabilities Found: XX

## Subdomains
- subdomain1.example.com
- subdomain2.example.com

## Vulnerability Findings
1. **XSS**: URL: `https://example.com/test?param=<script>alert()</script>`
2. **SQL Injection**: URL: `https://example.com?id=1'`
```

#### **JSON Output**

```python
import json

data = {
    "target": "example.com",
    "subdomains": ["sub1.example.com", "sub2.example.com"],
    "vulnerabilities": [
        {"type": "XSS", "url": "https://example.com/test"},
        {"type": "SQLi", "url": "https://example.com?id=1"}
    ]
}

with open("report.json", "w") as f:
    json.dump(data, f, indent=4)
```

***

### **8. Testing and Debugging**

#### **Unit Testing**

* Use Python’s `unittest` module:

  ```python
  import unittest

  def add(a, b):
      return a + b

  class TestMath(unittest.TestCase):
      def test_add(self):
          self.assertEqual(add(1, 2), 3)

  if __name__ == "__main__":
      unittest.main()
  ```

#### **Debugging Tips**

1. Use `set -x` in Bash scripts for tracing.
2. In Python, use `pdb`:

   ```python
   import pdb; pdb.set_trace()
   ```

***

### **9. Useful Libraries and Frameworks**

#### **Bash**

* [Httpx](https://github.com/projectdiscovery/httpx): Fast HTTP requests.
* [Nuclei](https://github.com/projectdiscovery/nuclei): Vulnerability scanner.

#### **Python**

* [Requests](https://docs.python-requests.org/): Simplified HTTP requests.
* [BeautifulSoup](https://www.crummy.com/software/BeautifulSoup/): Web scraping.

#### **Go**

* [Amass](https://github.com/owasp-amass): Subdomain enumeration.
* [FFuF](https://github.com/ffuf/ffuf): Fast web fuzzer.

***

### **10. Resources and Learning Platforms**

* [OWASP Cheat Sheets](https://cheatsheetseries.owasp.org/): Security best practices.
* [HackTricks](https://book.hacktricks.xyz/): Offensive techniques.
* [PayloadsAllTheThings](https://github.com/swisskyrepo/PayloadsAllTheThings): Exploitation payloads.

***

This comprehensive guide should give you the confidence to start building, automating, and refining your own tools. 🚀

Certainly! Here's an **expanded version** of the **Awesome List** with **additional tips, tools, workflows**, and techniques.

***

## **Extended Ultimate Awesome List for Tool Building and Automation**

> Build faster, more efficient tools for bug bounty, cybersecurity, and recon tasks with this comprehensive guide.

***

### **11. Advanced Recon Techniques**

#### **Subdomain Enumeration**

1. **Passive Techniques**:
   * Use `crt.sh` for Certificate Transparency Logs.
   * Tools: `subfinder`, `amass`, `assetfinder`, `dnsx`.
2. **Active Techniques**:
   * DNS brute-forcing with `puredns` or `dnsx`.
   * Permutation-based enumeration using `gotator`.
3. **DNS Data Gathering**:
   * Retrieve DNS records with `dig`, `host`, or `dnsx`:

     ```bash
     dnsx -d target.com -a -resp
     ```
4. **Recursive Subdomain Search**:
   * Use tools like `dsieve` to recursively find deeper subdomains:

     ```bash
     dsieve -d target.com -r
     ```

#### **URL Discovery**

* Combine **Wayback Machine**, **Common Crawl**, and **gau** for maximum coverage:

  ```bash
  cat targets.txt | gau | sort -u > urls.txt
  waybackurls < target.txt >> urls.txt
  ```

***

### **12. Dynamic Wordlist Generation**

1. **Generating Wordlists from JS Files**
   * Use `getjs` and `jsluice`:

     ```bash
     cat urls.txt | getjs | xargs -n 1 jsluice -u > wordlist.txt
     ```
2. **Generate Password Lists**
   * Use `pydictor`:

     ```bash
     pydictor -base rule.txt -o passwords.txt
     ```
3. **Custom Subdomain Lists**
   * Combine existing lists with `dnsvalidator`:

     ```bash
     cat resolvers.txt | dnsvalidator -tL subdomains.txt > valid.txt
     ```

***

### **13. Advanced Scripting Tips**

#### **Parallel Processing**

* Use GNU Parallel for running commands on multiple cores:

  ```bash
  cat subdomains.txt | parallel -j 10 "curl -Is {}"
  ```

#### **Tool Dependency Checker**

* Ensure all required tools are installed before execution:

  ```bash
  tools=("nuclei" "amass" "subfinder")
  for tool in "${tools[@]}"; do
    if ! command -v $tool &>/dev/null; then
      echo "$tool is not installed. Please install it first!"
      exit 1
    fi
  done
  ```

#### **Error Handling**

* Redirect errors to a separate log file:

  ```bash
  command_here >> results.log 2>> errors.log
  ```

***

### **14. Automation Pipelines**

#### **GitHub Actions Pipeline**

Automate recon with scheduled scans:

```yaml
name: Automated Recon Pipeline

on:
  schedule:
    - cron: '0 2 * * *' # Daily at 2 AM
  workflow_dispatch:

jobs:
  recon:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout Code
        uses: actions/checkout@v3

      - name: Install Tools
        run: |
          sudo apt update
          sudo apt install -y subfinder httpx nuclei

      - name: Run Recon
        run: |
          ./scripts/recon.sh
```

#### **Google Cloud Automation**

* Schedule automated scans with Google Cloud Functions and Cloud Scheduler.
* Example function for subdomain enumeration:

  ```python
  import subprocess

  def enumerate(event, context):
      domain = event.get("domain", "example.com")
      subprocess.run(["subfinder", "-d", domain, "-o", f"{domain}.txt"])
  ```

***

### **15. Post-Processing and Data Visualization**

#### **Filtering Unique URLs**

* Deduplicate and sort URLs:

  ```bash
  cat urls.txt | sort -u > clean_urls.txt
  ```

#### **Create Visual Maps**

1. **Subdomain Graphs**
   * Use `amass` and `Maltego` for visualizing subdomain connections.
   * Export with:

     ```bash
     amass viz -d target.com -o output.json
     ```
2. **Network Graphs**
   * Use tools like `neo4j` or `Graphviz`.

#### **HTML Reporting**

* Use Python libraries like `BeautifulSoup` or `j2html` to generate reports:

  ```python
  from jinja2 import Template

  template = Template("<h1>Report for {{ domain }}</h1>")
  print(template.render(domain="example.com"))
  ```

***

### **16. Tool Examples**

#### **API Token Validator**

```python
import requests

def check_api_key(api_key):
    response = requests.get("https://api.example.com", headers={"Authorization": f"Bearer {api_key}"})
    if response.status_code == 200:
        print("API Key is valid")
    else:
        print("Invalid API Key")
```

#### **Port Scanning Automation**

```bash
#!/bin/bash

target=$1
output_dir="./output/$target"

mkdir -p $output_dir

echo "[+] Scanning ports on $target..."
nmap -sC -sV -oN $output_dir/ports.txt $target
```

***

### **17. Notifications**

#### **Slack Integration**

Send results to Slack:

```bash
webhook_url="https://hooks.slack.com/services/your/webhook/url"
message="Recon complete for $target"

curl -X POST -H 'Content-type: application/json' --data '{"text":"'"$message"'"}' $webhook_url
```

#### **Telegram Bot**

Notify via Telegram:

```python
import requests

bot_token = "your_bot_token"
chat_id = "your_chat_id"
message = "Recon complete"

requests.post(f"https://api.telegram.org/bot{bot_token}/sendMessage", data={"chat_id": chat_id, "text": message})
```

***

### **18. Advanced Tools and Frameworks**

#### **Recon Tools**

* **katana**: Fast crawler for endpoints.
* **waymore**: Wayback URLs with custom filtering.
* **Interlace**: Automate tool chaining for multithreaded scans.

#### **Fuzzing Tools**

* **ffuf**: Fast web fuzzer for directories and parameters.
* **GoFuzz**: Fuzzing Go applications.

***

### **19. Continuous Improvement**

#### **Set Benchmarks**

* Track metrics for improvement:
  * Time to find vulnerabilities.
  * Tool efficiency (false positives vs. true positives).
  * Automation speed.

#### **Integrate Machine Learning**

* Use AI-based tools like `ChatGPT` or `Weka` to analyze recon data patterns for hidden vulnerabilities.

***

### **20. Resources for Inspiration**

#### **Books**

* **"Black Hat Python"** by Justin Seitz.
* **"Hacking APIs"** by Corey Ball.

#### **Communities**

* [Bugcrowd University](https://university.bugcrowd.com/)
* [OWASP Slack](https://owasp.slack.com/)

#### **Learning Platforms**

* [HackerOne's Directory](https://hackerone.com/directory)
* [TryHackMe Labs](https://tryhackme.com/)
* [Hack The Box](https://hackthebox.com/)

***

This extended guide is designed to give you **everything** you need to get started with **tool development, automation**, and **advanced recon workflows**. Here’s an extended **Awesome List** to further cover the essentials and advanced aspects of scripting, automation, and tool creation in bug hunting and cybersecurity. This builds on the previous list with **new categories and actionable insights**:

***

### **31. Web Scraping and Automation**

#### **Python Web Scraping**

1. **Scraping with `requests` and `BeautifulSoup`:**

   ```python
   import requests
   from bs4 import BeautifulSoup

   url = "https://example.com"
   response = requests.get(url)
   soup = BeautifulSoup(response.content, "html.parser")

   for link in soup.find_all("a"):
       print(link.get("href"))
   ```
2. **Scraping APIs**:

   ```python
   import requests

   url = "https://api.example.com/data"
   headers = {"Authorization": "Bearer your_api_token"}
   response = requests.get(url, headers=headers)
   print(response.json())
   ```

#### **Browser Automation**

1. **Using Selenium for Dynamic Content:**

   ```python
   from selenium import webdriver

   driver = webdriver.Chrome()
   driver.get("https://example.com")
   print(driver.page_source)
   driver.quit()
   ```
2. **Headless Browsing with Puppeteer:**

   ```javascript
   const puppeteer = require('puppeteer');
   (async () => {
       const browser = await puppeteer.launch({ headless: true });
       const page = await browser.newPage();
       await page.goto('https://example.com');
       console.log(await page.content());
       await browser.close();
   })();
   ```

***

### **32. Advanced Network Scanning**

#### **Bash Utilities**

1. **Masscan for Fast Port Scanning**:

   ```bash
   masscan -p1-65535 192.168.1.0/24 --rate 10000 -oG masscan_results.txt
   ```
2. **Custom Banner Grabbing**:

   ```bash
   nmap -sV --script=banner 192.168.1.1
   ```

#### **Python Scanners**

1. **Custom Port Scanner**:

   ```python
   import socket

   def scan(ip, port):
       try:
           sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
           sock.settimeout(1)
           sock.connect((ip, port))
           print(f"Port {port} is open on {ip}")
           sock.close()
       except:
           pass

   scan("192.168.1.1", 80)
   ```

***

### **33. OSINT Automation**

#### **OSINT Tools**

1. **Email Enumeration with `holehe`:**

   ```bash
   holehe -l emails.txt
   ```
2. **Search Leaked Credentials**:

   ```bash
   theharvester -d target.com -b all
   ```

#### **Custom Scripts**

1. **Automate Google Dorking:**

   ```bash
   dorks=("site:example.com inurl:admin" "site:example.com ext:sql")
   for dork in "${dorks[@]}"; do
       query=$(echo $dork | sed 's/ /+/g')
       curl "https://www.google.com/search?q=$query"
   done
   ```
2. **LinkedIn Scraping for Employee Info:**

   ```python
   import requests

   url = "https://www.linkedin.com/search/results/people/"
   headers = {"User-Agent": "Mozilla/5.0"}
   response = requests.get(url, headers=headers)
   print(response.text)
   ```

***

### **34. Advanced Vulnerability Exploitation**

#### **Exploitation Scripts**

1. **SQL Injection Exploitation with Python:**

   ```python
   import requests

   payload = "' OR '1'='1"
   url = "https://example.com/login"
   data = {"username": payload, "password": payload}
   response = requests.post(url, data=data)
   print(response.text)
   ```
2. **XSS Automation:**

   ```bash
   echo "<script>alert(1)</script>" > payloads.txt
   cat urls.txt | while read url; do
       curl "$url?q=$(cat payloads.txt)"
   done
   ```

***

### **35. API Security Testing**

#### **Custom Scripts**

1. **API Endpoint Testing**:

   ```python
   import requests

   url = "https://api.example.com/v1/data"
   headers = {"Authorization": "Bearer token"}
   response = requests.get(url, headers=headers)

   if response.status_code == 200:
       print("API is functional")
   else:
       print("API error:", response.status_code)
   ```
2. **Automate Rate-Limiting Checks**:

   ```bash
   for i in {1..100}; do
       curl -X GET "https://api.example.com" &
   done
   ```

***

### **36. Data Extraction**

#### **Extract Key Info**

1. **Find All IPs in a Log File**:

   ```bash
   grep -Eo '([0-9]{1,3}\.){3}[0-9]{1,3}' logs.txt | sort -u > ips.txt
   ```
2. **Extract Emails**:

   ```bash
   grep -Eo "\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b" logs.txt > emails.txt
   ```

***

### **37. Continuous Monitoring Pipelines**

#### **GitHub Actions for Automation**

1. **Recon Automation:**

   ```yaml
   name: Recon Automation
   on:
     schedule:
       - cron: '0 0 * * *'

   jobs:
     recon:
       runs-on: ubuntu-latest
       steps:
         - name: Subdomain Enumeration
           run: subfinder -d example.com > subs.txt
   ```
2. **Slack Alerts for Findings**:

   ```bash
   curl -X POST -H 'Content-type: application/json' --data '{"text":"Scan completed!"}' $SLACK_WEBHOOK
   ```

***

### **38. File Handling in Automation**

#### **Parse and Process Large Files**

1. **Bash: Split Large Files**:

   ```bash
   split -l 1000 large_file.txt small_
   ```
2. **Python: Process JSON**:

   ```python
   import json

   with open("data.json") as f:
       data = json.load(f)
       for item in data:
           print(item)
   ```

***

### **39. Advanced Reporting**

#### **Visualizations**

1. **Graph Vulnerabilities:**
   * Use Python’s `matplotlib`:

     ```python
     import matplotlib.pyplot as plt

     data = [10, 15, 20]
     labels = ["XSS", "SQLi", "SSRF"]

     plt.pie(data, labels=labels, autopct='%1.1f%%')
     plt.title("Vulnerability Distribution")
     plt.show()
     ```
2. **Heatmaps for Severity**:

   ```python
   import seaborn as sns
   import matplotlib.pyplot as plt

   data = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
   sns.heatmap(data, annot=True)
   plt.show()
   ```

***

### **40. Quick Automation Ideas**

#### **Instant Tools**

1. **Certificate Transparency**:

   ```bash
   curl -s "https://crt.sh/?q=example.com" | grep "example.com"
   ```
2. **Screenshot Script**:

   ```bash
   cat live.txt | while read url; do
       eyewitness -u $url
   done
   ```
3. **Check for WAF**:

   ```bash
   wafw00f https://example.com
   ```

***

This expanded list includes **even more actionable insights** for scripting, automation, and creating tools with **over 40 unique sections**. Each item is tailored to help you **create effective, automated, and scalable bug bounty workflows**.

***

### **21. Advanced Recon Pipelines**

#### **GitOps for Recon**

1. **Use GitLab CI/CD** for recon pipelines:

   ```yaml
   stages:
     - recon
     - scan

   recon:
     stage: recon
     script:
       - subfinder -d target.com > subs.txt
       - httpx -l subs.txt -o live.txt

   scan:
     stage: scan
     script:
       - nuclei -l live.txt -t ~/nuclei-templates/
   ```
2. **Kubernetes Integration**:
   * Deploy tools like `nuclei` and `ffuf` on a Kubernetes cluster for scalable recon.
3. **Axiom for Distributed Scanning**:
   * Create distributed pipelines using Axiom:

     ```bash
     axiom-scan live.txt -m nuclei -t ~/nuclei-templates/
     ```

***

### **22. Custom Scripting for Bug Bounty**

#### **Enhanced Bash Utilities**

1. **Retry Logic for Unstable Commands**:

   ```bash
   retry() {
       local n=1
       local max=5
       local delay=5
       while true; do
           "$@" && break || {
               if [[ $n -lt $max ]]; then
                   ((n++))
                   echo "Command failed. Attempt $n/$max:"
                   sleep $delay;
               else
                   echo "Command failed after $n attempts."
                   return 1
               fi
           }
       done
   }
   ```

   * Use as:

     ```bash
     retry curl -I https://example.com
     ```
2. **Dynamic Wordlist Updates**:

   ```bash
   cat live_urls.txt | grep ".js" | cut -d '/' -f3 | sort -u > js_wordlist.txt
   ```
3. **Automated Screenshot Script**:

   ```bash
   cat live.txt | aquatone -out screenshots
   ```

#### **Python Recon Scripts**

1. **HTTP Header Analyzer**:

   ```python
   import requests

   def analyze_headers(url):
       response = requests.get(url)
       headers = response.headers
       for header, value in headers.items():
           print(f"{header}: {value}")

   analyze_headers("https://example.com")
   ```
2. **Directory Brute-Forcer**:

   ```python
   import requests

   url = "https://example.com"
   wordlist = ["admin", "login", "config"]
   for word in wordlist:
       response = requests.get(f"{url}/{word}")
       if response.status_code == 200:
           print(f"Found: {url}/{word}")
   ```

***

### **23. Advanced Data Parsing**

#### **Extract Domains from JS Files**

1. **Using Bash**:

   ```bash
   grep -oP 'https?://[a-zA-Z0-9.-]+' *.js | sort -u > domains.txt
   ```
2. **With Python**:

   ```python
   import re

   with open("file.js", "r") as f:
       content = f.read()
       urls = re.findall(r"https?://[a-zA-Z0-9./-]+", content)
       for url in urls:
           print(url)
   ```

#### **Parse JSON Files for Sensitive Data**

* Use `jq`:

  ```bash
  jq '.keys[] | select(.type=="AWS")' sensitive.json
  ```

***

### **24. Enhanced Vulnerability Detection**

#### **Custom Scripts**

1. **SSRF Testing**:

   ```python
   import requests

   payload = {"url": "http://localhost:8080/admin"}
   response = requests.post("https://target.com/api", json=payload)
   print(response.text)
   ```
2. **XSS Payload Testing**:

   ```bash
   cat urls.txt | while read url; do
       curl "$url?q=<script>alert(1)</script>"
   done
   ```
3. **SQL Injection Automation**:

   ```bash
   sqlmap -u "https://example.com?id=1" --batch --dbs
   ```

***

### **25. Workflow Optimization Tools**

#### **Version Control**

* Use Git hooks to enforce standards:

  ```bash
  # pre-commit hook
  echo "Running security checks..."
  ```

#### **Linters and Formatters**

* Python: `black`, `flake8`
* Bash: `shellcheck`

#### **Automated Deployment**

* Use `Docker`:

  ```dockerfile
  FROM python:3.9
  WORKDIR /app
  COPY . .
  RUN pip install -r requirements.txt
  CMD ["python", "app.py"]
  ```

***

### **26. Best Practices for Tool Development**

#### **Modularization**

* Use reusable functions:

  ```python
  def subdomain_enum(domain):
      # Code here
      return results
  ```

#### **Error Handling**

* Provide detailed error messages:

  ```python
  try:
      response = requests.get("https://example.com")
      response.raise_for_status()
  except requests.RequestException as e:
      print(f"Error: {e}")
  ```

#### **Performance Optimization**

* Multi-threading:

  ```python
  from concurrent.futures import ThreadPoolExecutor

  def scan(url):
      print(f"Scanning {url}")

  urls = ["https://example.com", "https://test.com"]
  with ThreadPoolExecutor(max_workers=5) as executor:
      executor.map(scan, urls)
  ```

***

### **27. Continuous Monitoring**

#### **Scheduled Tasks**

* Use `cron` for periodic scans:

  ```bash
  0 2 * * * /path/to/recon.sh >> /path/to/logs.txt
  ```

#### **Real-Time Alerts**

* Integrate Slack or Telegram for notifications:

  ```bash
  curl -X POST -H 'Content-type: application/json' --data '{"text":"Scan completed"}' $SLACK_WEBHOOK
  ```

***

### **28. GitHub Awesome List Templates**

#### **Building Your Own List**

* Use [GitHub Markdown Templates](https://github.com/topics/awesome-list):
  * **Structure**:

    ```
    # Awesome List

    ## Introduction
    ## Tools
    ## Resources
    ## Tutorials
    ## Contributing
    ```

#### **Automated Updates**

* Use GitHub Actions to periodically update content:

  ```yaml
  on:
    schedule:
      - cron: '0 0 * * 1'
  ```

***

### **29. Advanced Reporting**

#### **HTML Reports**

* Create visually appealing reports using `ReportLab` or `Flask`:

  ```python
  from flask import Flask, render_template

  app = Flask(__name__)

  @app.route("/")
  def index():
      return render_template("report.html")
  ```

#### **Markdown Reports**

1. **Script Example**:

   ```bash
   echo "# Recon Report" > report.md
   cat results.txt >> report.md
   ```
2. **Export to PDF**:
   * Use `pandoc`:

     ```bash
     pandoc report.md -o report.pdf
     ```

***

### GitHub has blocked your push because it detected a secret (like a GitHub Personal Access Token) in your repository. To resolve this and safeguard your repository:

***

#### **Steps to Fix and Prevent Issues**

**1. Remove Secrets from History**

* **Identify the Secret:** GitHub tells you where the secret is located, e.g., `MyMac/Automated-Scanner/tools/.tokens:1`. Open the file and remove the secret.
* **Amend the Commit:** After removing the secret, re-commit the changes:

  ```bash
  git rm --cached MyMac/Automated-Scanner/tools/.tokens
  git commit --amend --no-edit
  git push origin main --force
  ```

**2. Use `git filter-repo` to Remove Secrets from Entire History**

If the secret exists in older commits:

* **Install `git-filter-repo`:**

  ```bash
  pip install git-filter-repo
  ```
* **Remove the Secret:** Replace `<path>` with the file path containing the secret:

  ```bash
  git filter-repo --path <path> --invert-paths
  ```
* **Force Push Clean History:**

  ```bash
  git push origin main --force
  ```

***

#### **Set Up Best Practices to Prevent Future Issues**

**3. Enable GitHub Push Protection**

GitHub will block pushes containing sensitive information by default if push protection is enabled. Ensure it's active in your repository settings:

1. Go to **Settings > Code Security and Analysis > Push Protection**.
2. Enable **Push Protection**.

**4. Use `.gitignore` to Prevent Secrets from Being Committed**

Add paths of sensitive files to a `.gitignore` file:

```plaintext
# Ignore token files
*.tokens
*.env
```

Then stage and commit the `.gitignore` file:

```bash
git add .gitignore
git commit -m "Add .gitignore to prevent committing sensitive files"
git push origin main
```

**5. Scan for Secrets Locally**

Use tools like [GitGuardian CLI](https://www.gitguardian.com/ggshield) to scan your commits for secrets before pushing:

```bash
ggshield secret scan pre-commit
```

**6. Replace Existing Tokens**

If a token was exposed, revoke it and create a new one:

* **Revoke the token** via GitHub's **Settings > Developer Settings > Personal Access Tokens**.
* **Create a new token** and securely store it in environment variables or secret management tools like AWS Secrets Manager or HashiCorp Vault.

**7. Use `.env` Files for Secrets**

Store secrets in `.env` files and load them dynamically into your scripts using tools like `dotenv`. Example `.env` file:

```plaintext
GITHUB_TOKEN=your_token_here
```

Add `.env` to `.gitignore`:

```plaintext
.env
```

***

#### **Push Again After Fixing Issues**

* After cleaning the history and ensuring no secrets remain, retry your push:

  ```bash
  git push origin main
  ```

#### **Comprehensive GitHub Solutions and Best Practices**

Here’s a detailed guide to handling common Git issues, maintaining repository hygiene, securing your code, and improving workflows for professional-grade repositories.

***

#### **1. General Git and GitHub Commands**

**Clone a Repository**

```bash
git clone https://github.com/username/repo.git
```

**Create and Push a New Branch**

```bash
git checkout -b feature-branch
git push origin feature-branch
```

**Merge Branches**

```bash
git checkout main
git merge feature-branch
git push origin main
```

**Stash Uncommitted Changes**

```bash
git stash
# Apply the stash later
git stash pop
```

***

#### **2. Fixing Common Git Issues**

**Remove Untracked Files**

```bash
git clean -f -d
```

**Undo the Last Commit**

```bash
git reset --soft HEAD~1  # Keeps changes staged
git reset --hard HEAD~1  # Discards changes completely
```

**Resolve Merge Conflicts**

1. Open conflicted files.
2. Edit the conflict markers (`<<<<<<`, `======`, `>>>>>>`).
3. Stage the resolved files:

   ```bash
   git add <file>
   ```
4. Continue the merge:

   ```bash
   git commit
   ```

**Fix Detached HEAD**

```bash
git checkout <branch>
```

***

#### **3. Managing Secrets**

**Prevent Secrets in Commits**

Add sensitive files to `.gitignore`:

```plaintext
.env
*.key
*.pem
*.tokens
```

**Scan Commits for Secrets**

Use tools like [TruffleHog](https://github.com/trufflesecurity/trufflehog):

```bash
trufflehog --regex --entropy=True https://github.com/username/repo.git
```

**Replace Leaked Secrets**

If a secret is exposed:

1. **Revoke the token** immediately.
2. Use `git filter-repo` to remove it:

   ```bash
   git filter-repo --path-sensitive-file --invert-paths
   ```
3. Push the cleaned history:

   ```bash
   git push origin main --force
   ```

***

#### **4. Improving Repository Security**

**Enable 2FA for Your GitHub Account**

1. Go to **Settings > Security > Two-factor authentication**.
2. Follow the prompts to set up 2FA.

**Set Up Branch Protection**

1. Go to **Settings > Branches > Add branch protection rule**.
2. Enable:
   * Require pull request reviews.
   * Require status checks.
   * Restrict who can push.

**Enable Secret Scanning**

1. Go to **Settings > Security & Analysis**.
2. Enable **Secret Scanning** and **Push Protection**.

***

#### **5. Automating Workflows**

**Using GitHub Actions**

Create a `.github/workflows/main.yml` file:

```yaml
name: CI/CD Pipeline

on:
  push:
    branches:
      - main

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout Code
        uses: actions/checkout@v3

      - name: Run Tests
        run: |
          npm install
          npm test
```

**Schedule Automated Tasks**

Run a workflow daily:

```yaml
on:
  schedule:
    - cron: "0 0 * * *"
```

***

#### **6. Commit Hygiene**

**Write Descriptive Commit Messages**

Follow the conventional commit format:

```plaintext
<type>(<scope>): <short summary>

[optional body]

[optional footer(s)]
```

Example:

```plaintext
feat(auth): add login endpoint
fix(auth): resolve token expiration issue
```

**Sign Commits**

Enable GPG signing for commits:

```bash
git config --global user.signingkey <GPG_KEY_ID>
git config --global commit.gpgsign true
```

***

#### **7. Version Control Best Practices**

**Tagging Releases**

```bash
git tag -a v1.0.0 -m "Initial release"
git push origin v1.0.0
```

**Changelog Automation**

Use [Conventional Changelog](https://github.com/conventional-changelog):

```bash
npx conventional-changelog-cli -p angular
```

***

#### **8. Collaboration**

**Create a Pull Request**

1. Push changes:

   ```bash
   git push origin feature-branch
   ```
2. Go to the repository on GitHub and create a PR.

**Review PRs**

* Use GitHub’s review tools to add comments.
* Mark a PR as approved or request changes.

***

#### **9. Repository Hygiene**

**Archive Old Branches**

Delete merged branches:

```bash
git branch -d feature-branch
git push origin --delete feature-branch
```

**Clean Up Commit History**

Use rebase for a cleaner history:

```bash
git rebase -i HEAD~<number-of-commits>
```

***

#### **10. Advanced Git Commands**

**Revert a Specific Commit**

```bash
git revert <commit-hash>
```

**Cherry-Pick a Commit**

```bash
git cherry-pick <commit-hash>
```

**View Commit History**

```bash
git log --oneline --graph --decorate
```

**Show Changes Between Branches**

```bash
git diff branch1..branch2
```

***

#### **11. Backup and Restore**

**Backup a Repository**

```bash
git bundle create repo.bundle --all
```

**Restore from Backup**

```bash
git clone repo.bundle
```

***

#### **12. Troubleshooting**

**Fix Corrupted Repositories**

```bash
git fsck --full
```

**Rebuild the Index**

```bash
rm -f .git/index
git reset
```

**Recover Lost Commits**

```bash
git reflog
git reset --hard <commit-hash>
```

***

#### **13. Resources for Learning**

* [Pro Git Book](https://git-scm.com/book/en/v2)
* [GitHub Docs](https://docs.github.com/)
* [Git Cheatsheet](https://education.github.com/git-cheat-sheet-education.pdf)

***

This list ensures you have everything covered, from basic Git operations to advanced GitHub features, to help you create efficient, secure, and collaborative workflows. If you still encounter issues, GitHub’s documentation [here](https://docs.github.com/code-security/secret-scanning/working-with-secret-scanning-and-push-protection/working-with-push-protection-from-the-command-line#resolving-a-blocked-push) provides more details on resolving blocked pushes.

### **30. Learning Resources**

#### **Tutorials**

1. [The Bug Hunter’s Methodology](https://hackerone.com/blog/bug-hunter-methodology)
2. [OWASP Top 10 Vulnerabilities](https://owasp.org/www-project-top-ten/)

#### **Videos**

* YouTube Channels:
  * [LiveOverflow](https://www.youtube.com/c/LiveOverflow)
  * [NahamSec](https://www.youtube.com/c/NahamSec)

#### **Books**

1. **"The Web Application Hacker's Handbook"** by Dafydd Stuttard.
2. **"Practical Binary Analysis"** by Dennis Andriesse.

***

This expanded Awesome List now provides **200+ actionable tips, tools, and workflows** for building, automating, and optimizing cybersecurity scripts and tools!