New Import Tool: External Content Sync

External Content Sync Dashboard

Our new External Content Sync tool streamlines the process of importing external data into RebelMouse. This system allows us to seamlessly map external JSON data to RebelMouse models and import it as posts, sections, authors, and media. The new system is significantly faster than before and unlike previous methods, a failed record, will not cancel the entire import. Instead, failures are logged in a concise, human-readable, and actionable format for easy troubleshooting. This tool only supports JSON format. Any other data format must be converted to JSON before import.

Key Components

1. Frontend Dashboard
   • Users can upload raw JSON data, parse it, split it into chunks, and send it to our API.
   • Provides tools to manage and start imports efficiently.
2. REST API
   • Responsible for consuming raw data, managing import-related tasks, and generating import feeds for the processing worker.
3. Client Backends
   • Supports data mapping before storing it in RebelMouse, assisting in preliminary transformation steps.

Import Local Data

Upload Files for Import

The Upload File option allows users to import external data seamlessly. On this page, you will find the following fields:

1. Entity Type Dropdown – Select the type of data being imported, such as Posts, Sections, Assets, Authors, and Redirects.
2. File Key – This identifier allows you to reuse mappings across multiple files of the same entity.
   • Example: Both my_posts and my_products can map to Posts but use different mappers.
3. Unique Identifier of the Record –
   • This field is required for all records and helps identify them internally.
   • Any record without a unique identifier will be rejected.
   • The value must be a string or a number.
4. Choose File Button – Select the local JSON file for import.
   • The file must be a JSON array. Convert other formats before uploading.
   • Each file must contain a single entity (e.g., if posts contain section objects, split them and link by ID).
   • If using multiple files, ensure they are slices or chunks of a larger file to maintain data consistency.
5. Submit button - Once all required fields are filled and the data is successfully added, click the Submit button to begin processing the import.

CSV to JSON Converter

Convert CSV files to JSON before importing. Click Choose File, then Submit to upload. The converted JSON will download automatically.

Split Large JSON Files

Upload a large JSON file (max 400MB per chunk), click Submit, and download the split versions[chunk] automatically.

Cloud Data

Manage Raw Files

View and track the latest uploaded files and their records.

Manage Extractor

Handles preliminary data creation before final import.

Manage Mapping

Define mappings for posts, sections, authors, etc. Provide mapping URLs to client backends for data conversion. Also, list domains to replace external ones with RebelMouse domains.

How Mapping Works?

Client backend scripts process import data, transforming it into a RebelMouse-compatible format. For example, date objects are converted into timestamps before import.

In clientbackend scripts, we reference the file location to identify entities using unique identifiers. The importer then checks and maps entities accordingly.

• The script iterates over the data, applying mappings through references.
• Each setup requires different backend scripts since file mappings may vary.
• Adjustments to the script ensure accurate data transformation.

Manage Preliminary Content

View and manage preliminary data for posts, authors, sections, assets, and redirects before it’s consumed by RebelMouse. Check logs for errors and ensure data accuracy.

Actions for Preliminary Data

View File

This is the information we have saved for this record, ready to be consumed by RebelMouse

View logs

Logs for every operation of the system.

View Logs for all the errors

View Meta for the Error on Logs

Check error details through the View Meta to understand and troubleshoot issues.