feat: added a way to remap key name v18

README.md

@@ -1,16 +1,15 @@
-# Firestore / Firebase Typesense Search Extension ⚡ 🔍 
+# Firestore / Firebase Typesense Search Extension ⚡ 🔍
 
-A Firebase extension to sync data from your Firestore collection to [Typesense](https://typesense.org/), 
+A Firebase extension to sync data from your Firestore collection to [Typesense](https://typesense.org/),
 to be able to do full-text fuzzy search on your Firestore data, with typo tolerance, faceting, filtering, sorting, curation, synonyms, geosearch and more.
 
-This extension listens to your specified Firestore collection and syncs Firestore documents to Typesense 
+This extension listens to your specified Firestore collection and syncs Firestore documents to Typesense
 on creation, updates and deletes. It also provides a function to help you backfill data.
 
 **What is Typesense?**
 
 If you're new to [Typesense](https://typesense.org), it is an open source search engine that is simple to use, run and scale, with clean APIs and documentation. Think of it as an open source alternative to Algolia and an easier-to-use, batteries-included alternative to ElasticSearch. Get a quick overview from [this guide](https://typesense.org/docs/guide).
 
-
 ## ⚙️ Usage
 
 ### Step 1️⃣ : Setup Prerequisites
@@ -19,12 +18,12 @@ Before installing this extension, make sure that you have:
 
 1. [Set up a Cloud Firestore database](https://firebase.google.com/docs/firestore/quickstart) in your Firebase project.
 2. [Set up](https://typesense.org/docs/guide/install-typesense.html) a Typesense cluster on [Typesense Cloud](https://cloud.typesense.org) or [Self-Hosted](https://typesense.org/docs/guide/install-typesense.html#option-2-local-machine-self-hosting) (free).
-3. Set up a Typesense Collection either through the Typesense Cloud dashboard or 
-  through the [API](https://typesense.org/docs/latest/api/collections.html#create-a-collection).
-  
-⚠️ ☝️ #3 above is a commonly missed step. This extension **does not create the Typesense Collection for you**. Instead it syncs data to a Typesense collection you've already created. If you see an HTTP 404 in the extension logs, it's most likely because of missing this step. 
+3. Set up a Typesense Collection either through the Typesense Cloud dashboard or
+   through the [API](https://typesense.org/docs/latest/api/collections.html#create-a-collection).
+
+⚠️ ☝️ #3 above is a commonly missed step. This extension **does not create the Typesense Collection for you**. Instead it syncs data to a Typesense collection you've already created. If you see an HTTP 404 in the extension logs, it's most likely because of missing this step.
 
-### Step 2️⃣ : Install the Extension 
+### Step 2️⃣ : Install the Extension
 
 You can install this extension either through the Firebase Web console or through the Firebase CLI.
 
@@ -47,22 +46,23 @@ firebase ext:install typesense/firestore-typesense-search --project=[your-projec
 When you install this extension, you'll be able to configure the following parameters:
 
 | Parameter                   | Description                                                                                                                                                                                                                                                                                    |
-|-----------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| --------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
 | Firestore Collection Path   | The Firestore collection that needs to be indexed into Typesense.                                                                                                                                                                                                                              |
 | Firestore Collection Fields | A comma separated list of fields that need to be indexed from each Firestore document. Leave blank to index all fields.                                                                                                                                                                        |
+| Typesense Fields Renames    | A comma separated list of field renames in the format `old_field_name:new_field_name`. This is useful when you want to rename fields from Firestore to a different name in Typesense. Leave blank to not rename any fields.                                                                                                                                                                        |
 | Flatten Nested Documents    | Should nested documents in Firestore be flattened before they are indexed in Typesense? Set to "Yes" for Typesense Server versions v0.23.1 and below, since indexing Nested objects is natively supported only in Typesense Server v0.24 and above.                                            |
-| Typesense Hosts             | A comma-separated list of Typesense Hosts (only ___domain without https or port number). For single node clusters, a single hostname is sufficient. For multi-node Highly Available or (Search Delivery Network) SDN Clusters, please be sure to mention all hostnames in a comma-separated list. | 
+| Typesense Hosts             | A comma-separated list of Typesense Hosts (only ___domain without https or port number). For single node clusters, a single hostname is sufficient. For multi-node Highly Available or (Search Delivery Network) SDN Clusters, please be sure to mention all hostnames in a comma-separated list. |
 | Typesense API Key           | A Typesense API key with admin permissions. Click on "Generate API Key" in cluster dashboard in Typesense Cloud.                                                                                                                                                                               |
 | Typesense Collection Name   | Typesense collection name to index data into (you need to create this collection in Typesense yourself. This extension does not create the Typesense Collection for you).                                                                                                                      |
 | Cloud Functions ___location    | Where do you want to deploy the functions created for this extension? You usually want a ___location close to your database. For help selecting a ___location, refer to the [___location selection guide](https://firebase.google.com/docs/functions/locations).                                        |
 
 > ⚠️ You'll notice that there is no way to configure the port number or protocol.
-This is because this extension only supports connecting to Typesense running HTTPS on Port 443, since your data goes from Firebase to Typesense over the public internet and we want your data to be encrypted in transit.
-For Typesense Cloud, HTTPS is already configured for you.
-> 
+> This is because this extension only supports connecting to Typesense running HTTPS on Port 443, since your data goes from Firebase to Typesense over the public internet and we want your data to be encrypted in transit.
+> For Typesense Cloud, HTTPS is already configured for you.
+>
 > When self-hosting Typesense, you want to make sure you set `--api-port=443` and also get an SSL certificate from say [LetsEncrypt](https://letsencrypt.org/) or any registrar
-and configure Typesense to use it using the `--ssl-certificate` and `--ssl-certificate-key` [server parameters](https://typesense.org/docs/latest/api/server-configuration.html).
-> Alternatively, if you're running Typesense on your local machine, you can also set up a local HTTPS tunnel using something like [ngrok](https://ngrok.com/) (`ngrok http 8108`) and use the ngrok hostname in the extension. 
+> and configure Typesense to use it using the `--ssl-certificate` and `--ssl-certificate-key` [server parameters](https://typesense.org/docs/latest/api/server-configuration.html).
+> Alternatively, if you're running Typesense on your local machine, you can also set up a local HTTPS tunnel using something like [ngrok](https://ngrok.com/) (`ngrok http 8108`) and use the ngrok hostname in the extension.
 
 ##### Example
 
@@ -90,16 +90,15 @@ This will trigger the backfill background Cloud function, which will read data f
 
 ## ☁️ Cloud Functions
 
-* **indexToTypesenseOnFirestoreWrite:** A function that indexes data into Typesense when it's triggered by Firestore changes.
-
-* **backfillToTypesenseFromFirestore:** A function that backfills data from a Firestore collection into Typesense, triggered when a Firestore document with the path `typesense_sync/backfill` has the contents of `trigger: true`.
+- **indexToTypesenseOnFirestoreWrite:** A function that indexes data into Typesense when it's triggered by Firestore changes.
 
+- **backfillToTypesenseFromFirestore:** A function that backfills data from a Firestore collection into Typesense, triggered when a Firestore document with the path `typesense_sync/backfill` has the contents of `trigger: true`.
 
 ## 🔑 Access Required
 
 This extension will operate with the following project IAM roles:
 
-* datastore.user (Reason: Required to backfill data from your Firestore collection into Typesense)
+- datastore.user (Reason: Required to backfill data from your Firestore collection into Typesense)
 
 ## 🧾 Billing
 
@@ -112,7 +111,6 @@ To install an extension, your project must be on the [Blaze (pay as you go) plan
 - Usage of this extension also requires you to have a running Typesense cluster either on Typesense Cloud or some
   self-hosted server. You are responsible for any associated costs with these services.
 
-
 ## Development Workflow
 
 #### Run Emulator
@@ -147,10 +145,9 @@ firebase ext:info ./ --markdown > README.md
 - Update version number in extension.yaml
 - Add entry to CHANGELOG.md
 - Create release in GitHub
-- 
-    ```shell
-    firebase ext:dev:upload typesense/firestore-typesense-search
-    ```
+- ```shell
+  firebase ext:dev:upload typesense/firestore-typesense-search
+  ```
 
 ## ℹ️ Support
 

extension.yaml

@@ -73,6 +73,15 @@ params:
     example: field1,field2,field3
     default: ""
     required: false
+  - param: TYPESENSE_FIELDS_RENAMES
+    label: Typesense Fields Renames
+    description: >-
+      A comma separated list of field renames in the format `old_field_name:new_field_name`.
+      This is useful when you want to rename fields from Firestore to a different name in Typesense.
+      Leave blank to not rename any fields.
+    example: field1:field_1,field2:field_2,field3:field_3
+    default: ""
+    required: false
   - param: TYPESENSE_HOSTS
     label: Typesense Hosts
     description: >-
@@ -98,7 +107,7 @@ params:
   - param: FLATTEN_NESTED_DOCUMENTS
     label: Flatten Nested Documents
     description: >-
-      Should nested documents in Firestore be flattened by this extension before they are indexed in Typesense? 
+      Should nested documents in Firestore be flattened by this extension before they are indexed in Typesense?
       Set to "Yes" for Typesense versions 0.23.1 and earlier. Set to "No" for Typesense versions 0.24.0 and later.
     type: select
     options:

extensions/firestore-typesense-search.env.local

@@ -1,6 +1,7 @@
 LOCATION=us-central1
 FIRESTORE_COLLECTION_PATH=books
 FIRESTORE_COLLECTION_FIELDS=author,title,rating,isAvailable,___location,createdAt,nested_field,tags,nullField,ref
+TYPESENSE_FIELDS_RENAMES=id:book_id
 FLATTEN_NESTED_DOCUMENTS=false
 TYPESENSE_HOSTS=localhost
 TYPESENSE_PORT=8108

extensions/test-params-flatten-nested-true.local.env

@@ -1,6 +1,7 @@
 LOCATION=us-central1
 FIRESTORE_COLLECTION_PATH=books
 FIRESTORE_COLLECTION_FIELDS=author,title,rating,isAvailable,___location,createdAt,nested_field,tags,nullField,ref
+TYPESENSE_FIELDS_RENAMES=id:book_id
 FLATTEN_NESTED_DOCUMENTS=true
 TYPESENSE_HOSTS=localhost
 TYPESENSE_PORT=8108

extensions/test-params.example.env

@@ -1,6 +1,7 @@
 LOCATION=us-central1
 FIRESTORE_COLLECTION_PATH=books
 FIRESTORE_COLLECTION_FIELDS=author,title
+TYPESENSE_FIELDS_RENAMES=id:book_id
 FLATTEN_NESTED_DOCUMENTS=true
 TYPESENSE_HOSTS=xxx-1.a1.typesense.net
 TYPESENSE_COLLECTION_NAME=books

functions/src/config.js

@@ -5,6 +5,15 @@ module.exports = {
         .split(",")
         .map((f) => f.trim())
         .filter((f) => f),
+  typesenseFieldsRenames: (process.env.TYPESENSE_FIELDS_RENAMES || "")
+      .split(",")
+      .map((f) => f.trim())
+      .filter((f) => f)
+      .map((f) => {
+        const [from, to] = f.split(":").map((f) => f.trim());
+        return {from, to};
+      })
+      .filter((f) => f.from && f.to),
   shouldFlattenNestedDocuments: process.env.FLATTEN_NESTED_DOCUMENTS === "true",
   typesenseHosts:
     (process.env.TYPESENSE_HOSTS || "").split(",").map((e) => e.trim()),

functions/src/utils.js

@@ -18,6 +18,11 @@ const mapValue = (value) => {
   }
 };
 
+const mapKey = (key) => {
+  const newKey = config.typesenseFieldsRenames[key] || key;
+  return newKey;
+};
+
 /**
  * @param {DocumentSnapshot} firestoreDocumentSnapshot
  * @param {Array} fieldsToExtract
@@ -37,7 +42,7 @@ exports.typesenseDocumentFromSnapshot = async (
   }
 
   // Build a document with just the fields requested by the user, and mapped from Firestore types to Typesense types
-  const mappedDocument = Object.fromEntries(entries.map(([key, value]) => [key, mapValue(value)]));
+  const mappedDocument = Object.fromEntries(entries.map(([key, value]) => [mapKey(key), mapValue(value)]));
 
   // using flat to flatten nested objects for older versions of Typesense that did not support nested fields
   // https://typesense.org/docs/0.22.2/api/collections.html#indexing-nested-fields

test/backfillToTypesenseFromFirestore.spec.js

@@ -43,6 +43,7 @@ describe("backfillToTypesenseFromFirestore", () => {
         author: "Author A",
         title: "Title X",
         country: "USA",
+        id: "123",
       };
       const firestoreDoc = await firestore.collection(config.firestoreCollectionPath).add(book);
       // Wait for firestore cloud function to write to Typesense
@@ -76,6 +77,7 @@ describe("backfillToTypesenseFromFirestore", () => {
         id: firestoreDoc.id,
         author: book.author,
         title: book.title,
+        book_id: book.id,
       });
     });
   });
@@ -89,6 +91,7 @@ describe("backfillToTypesenseFromFirestore", () => {
           author: "Author A",
           title: "Title X",
           country: "USA",
+          id: "123",
         };
         const firestoreDoc = await firestore.collection(config.firestoreCollectionPath).add(book);
         // Wait for firestore cloud function to write to Typesense
@@ -125,6 +128,7 @@ describe("backfillToTypesenseFromFirestore", () => {
           id: firestoreDoc.id,
           author: book.author,
           title: book.title,
+          book_id: book.id,
         });
       });
     });
@@ -136,6 +140,7 @@ describe("backfillToTypesenseFromFirestore", () => {
           author: "Author A",
           title: "Title X",
           country: "USA",
+          id: "123",
         };
         await firestore.collection(config.firestoreCollectionPath).add(book);
         // Wait for firestore cloud function to write to Typesense

test/indexToTypesenseOnFirestoreWrite.spec.js

@@ -517,4 +517,39 @@ describe("indexToTypesenseOnFirestoreWrite", () => {
 
     expect(typesenseDocsStr).toBe("");
   });
+
+  it("indexed fields are renamed to match the configured field name mapping", async () => {
+    const docData = {id: "123"};
+
+    // create document in Firestore
+    const docRef = await firestore.collection(config.firestoreCollectionPath).add(docData);
+
+    // wait for the Firestore cloud function to write to Typesense
+    await new Promise((r) => setTimeout(r, 2500));
+
+    // check that the document was indexed
+    let typesenseDocsStr = await typesense
+        .collections(encodeURIComponent(config.typesenseCollectionName))
+        .documents()
+        .export();
+    const typesenseDocs = typesenseDocsStr.split("\n").map((s) => JSON.parse(s));
+
+    expect(typesenseDocs.length).toBe(1);
+    expect(typesenseDocs[0]).toStrictEqual({"id": docRef.id});
+    expect(typesenseDocs[0]).toStrictEqual({"book_id": docData.id});
+
+    // delete document in Firestore
+    await docRef.delete();
+
+    // wait for the Firestore cloud function to write to Typesense
+    await new Promise((r) => setTimeout(r, 2500));
+
+    // check that the document was deleted
+    typesenseDocsStr = await typesense
+        .collections(encodeURIComponent(config.typesenseCollectionName))
+        .documents()
+        .export();
+
+    expect(typesenseDocsStr).toBe("");
+  });
 });