Trouble in Squaradise

Sometime around 1:30am after hours of bashing website bugs, we ran into a problem we couldn’t figure out. Suddenly, the registry wasn’t working and was returning vague 403 errors in the console when we tried to complete a payment. I had been hot-fixing some bugs and deploying straight to the website so I tried to roll a few of these changes back with no success. My brother had a hunch that we had triggered some sort of fraud detection with our testing and prompted me to log in to Square, where sure enough I had a notice that my account required “additional information” in order to continue. I filled out the information they had requested from me, but in the morning I woke up to the following email:

I feel like I do actually understand this response, to an extent – I wasn’t operating a business, and this use case surely didn’t fall into any normal pattern Square would care to prioritize. I was still pretty disappointed by this turn, especially considering the swiftness of the action and the amount of work I had put in to integrate with Square over another platform. Most of all, I was incredibly thankful for all the time we spent testing this integration to trigger this issue before the website went live. I had done extensive testing myself prior to our bug-bashing weekend and was pretty sure we were good to go. I can’t imagine how much worse this would have been if guests had started to use Square to actually buy wedding gifts before they decided on an inquisition and account closure.

So, I went back to Google to find another payment processor, this time making sure that they would support my weird use case. Despite how great it was to have the full Square interface for managing inventory without changing the code, I was pretty burnt out and unwilling to rewrite all 600+ lines of code I wrote to integrate completely with the Square platform and dynamically retrieve the items and inventory from their system (I also had a policy against using AI to write any for this project – authenticity!). So, I opted for a much simpler approach where the inventory was hardcoded in the website client-side, and PayPal was used to process payments for the selected items.

export const REGISTRY_ITEMS: RegistryItem[] = [
    {
        id: 'checked-bags',
        title: 'Checked Bags',
        cost: 50,
        available: 4,
        image: 'checked-bags.jpeg',
        description: "...",
    },
    // ...
]

return <SimpleGrid>
    { REGISTRY_ITEMS.map(item => (
        <RegistryItem key={item.id} item={item} />
    ))}
</SimpleGrid>

While this wouldn’t be a viable approach for purchasing real items, since it essentially would allow a technically savvy user to name their own price during check out, it was good enough for our use case where we were simply accepting gifts from website users. If anyone wanted to “cheat the system” and give a gift for less money than it “cost,” that would be fine by me.

So, we were back in business with PayPal and I sent the updated website off to my brother to pressure-test while I worked on an automated test suite.

Integration Testing and Canaries

As overkill as it might be to have a canary running automated tests against your wedding website, I did have a deep fear that something would go wrong and nobody would be able to RSVP to our wedding due to some bug in my code. And, I felt this from-scratch challenge wouldn’t be truly complete until I implemented everything I would actually implement for a production system I would be comfortable delivering to a client, which definitely involves testing and monitoring in production. I wanted to be the first one to know if and when I broke something with a change, or something out of my control like the Square debacle happened again.

I put together a Lambda that used the Playwright test framework to interact with the wedding website in production and exercise all of the functionality I cared deeply about working, especially RSVPing. Although playwright does support both UI testing as well as raw API tests, I opted to just write the UI tests since I didn’t care as much about direct usages of the API in this case. I figured that any usage that would be exercised through my testing of the website frontend would cover what users would care about.

This is the part where I could have used some lessons from my prior engineering experience a bit better. Every time I have written code that is meant to automate web browser usage running on Lambda, I feel as though I have directly accelerated the progress of my male-pattern balding. I may be naive in this area of software development, but there is absolutely no reason that this should be so difficult. Doing this months out from my wedding and pulling my hair out trying to get every one of the dependencies and Chrome command line arguments absolutely correct so that everything plays nicely together in a headless, serverless environment (where you can’t test the behavior until you deploy to that environment!!) seems like an exercise in pure insanity.

In order to get everything reliably working, I ended up writing a lot of boilerplate wrapper logic for the tests that would capture screenshots and logs of any of the test failures and upload them to an S3 bucket so I could understand what had occurred remotely.

Through an extremely long and arduous process of trial, error, and inventing new swear words, I was finally able to get the tests reliably working, and set up an EventBridge rule to run them every 10 minutes and publish metrics for their results. Here’s the lambda function code I ended up with, which mostly, usually, works, although is not nearly as clean and locally-testable as I was envisioning when I started writing the playwright tests.

Metrics and logs

In addition to the canary metrics and metrics emitted by the server, I also created an API endpoint for the website to post metrics and logs from the client’s perspective so that I could have a full view into the performance of the website, and be alerted of any issues I needed to know about. The website creates a singleton that collects metrics and logs posted from each component, and then peridocally flushes them in the background to the server with an API call:

import {Client, Event, Metric} from "wedding-client";

export interface Metrics {
    postEvent(event: Event): void;
    postMetric(metric: Metric): void;
}

export class MetricsPublisher implements Metrics {
    private readonly MAX_METRICS_TO_POST = 100;
    private readonly POST_METRICS_INTERVAL = 10 * 1000;

    private readonly client: Client;
    private readonly invitationId: string;
    private pendingMetrics: Metric[] = [];
    private pendingEvents: Event[] = [];
    private started: Boolean = false;
    private interval?: number = undefined;

    constructor(client: Client, invitationId: string) {
        this.client = client;
        this.invitationId = invitationId;
    }

    start() {
        if (!this.started) {
            this.interval = window.setInterval(
                (function(scope) {
                    return function(){
                        scope.flushMetrics();
                    };
                })(this),
                this.POST_METRICS_INTERVAL
            );
        }
    }

    postEvent(event: Event) {
        this.pendingEvents.push(event);
    }

    postMetric(metric: Metric) {
        this.pendingMetrics.push(metric);
    }

    getMetricsToPost = () => {
        const metricsToPost: Metric[] = [];
        const numMetrics = Math.min(this.pendingMetrics.length, this.MAX_METRICS_TO_POST);
        for (let i = 0; i < numMetrics; i++) {
            const metric = this.pendingMetrics.shift();
            if (metric) {
                metricsToPost.push(metric);
            }
        }
        return metricsToPost;
    }

    getEventsToPost = () => {
        const eventsToPost: Event[] = [];
        const numEvents = Math.min(this.pendingEvents.length, this.MAX_METRICS_TO_POST);
        for (let i = 0; i < numEvents; i++) {
            const event = this.pendingEvents.shift();
            if (event) {
                eventsToPost.push(event);
            }
        }
        return eventsToPost;
    }

    flushMetrics() {
        const metricsToPost = this.getMetricsToPost();
        const eventsToPost = this.getEventsToPost();
        if (metricsToPost.length > 0 || eventsToPost.length > 0) {
            if (window.location.host.startsWith('localhost')) {
                // Don't post metrics when running locally
                console.log('would have posted metrics ', metricsToPost);
                console.log('would have posted events ', eventsToPost);
            } else {
                this.client.postMetrics({
                    invitationId: this.invitationId,
                    metrics: metricsToPost,
                    events: eventsToPost,
                }).catch(error => {
                    console.error('error posting metrics', error);
                });
            }
        }
    };
}

On the server side, I used the @aws-lambda-powertools/metrics library to post metrics emitted by the server or from the client to CloudWatch using the embedded CloudWatch Logs format, to avoid synchronous latency added by calling CloudWatch APIs as part of the request. As part of this process, I discovered that the CloudWatch metrics aggregations worked a bit differently than the model I had in my head – what I really wanted was the ability to alarm on a top-level aggregate metric, such as unhandled errors on any page on the website. If alerted, I wanted the ability to break that metric down to a list of unhandled error metrics for each page, to quickly determine what page was at issue before turning to the logs.

I had actually thought that CloudWatch would do this sort of dimension aggregation automatically, but that turned out not to be true, so to achieve this I wrote a wrapper around the metrics that would automatically aggregate any metric into an ALL dimension, to achieve a top-level view:

public static closeMetrics(requestId: string) {
    // Create the shared metrics instance for originals + aggregates
    const metrics = new Metrics({
        namespace: 'wedding_website',
        serviceName: 'server',
    });
    // Add the original dimensions for the metric
    metrics.addDimensions(this.metricDimensions[requestId]);
    // Add the original metric values
    MetricsProvider.metricsToPublish[requestId].forEach(metric => {
        metrics.addMetric(metric.name, metric.unit, metric.value, metric.resolution);
    })
    // Publish the original metrics with their dimensions
    metrics.publishStoredMetrics();
    metrics.clearDimensions();

    // Publish aggregates by setting every key of the original dimension to a value of ALL
    Object.keys(this.metricDimensions[requestId]).forEach(dimension => {
        metrics.addDimension(dimension, 'ALL');
    });
    // Add the original metric values
    MetricsProvider.metricsToPublish[requestId].forEach(metric => {
        metrics.addMetric(metric.name, metric.unit, metric.value, metric.resolution);
    })
    // Publish the aggregated metrics with ALL dimensions
    metrics.publishStoredMetrics();
    metrics.clearDimensions();
}

This aggregation was expensive, but I felt it was worth it. I ended up with about 104 unique metrics across the server, website, and canary which, as I mentioned earlier, contributed to about 90% of the cost of this project.

In addition to the metrics, I wanted a way to easily see what was happening across my website that didn’t involve diving into the lambda log streams, such as the distinct Events emitted by the frontend client. I created another DynamoDB facet called an AuditLog which I wrote to with a record of each mutating API call, and with events submitted from the clients:

{
 "LogId": "MIJMNB7LU0OOAL",
 "LogType": "RSVP_VALIDATION_ERROR",
 "Message": "Website displayed a validation error",
 "ResourceId": "MIJMMG5ACYH5U9",
 "ResourceType": "INVITATION",
 "EventData": {
  "errors": "At least one guest must be attending, otherwise please RSVP no."
 },
 "LogDay": "2025-11-29",
 "Timestamp": 1764380580207,
}

This would help me quickly notice any common issues that my guests were encountering and fix them proactively. As one example of where this came in useful, I noticed that there were a number of errors guests were encountering when manually logging in, and from the logs it was clear the reason why was that they were using their full house address instead of just the number (for example entering 123 Main St. instead of just 123), which would cause the guest lookup to fail. I was able to fix this common issue by just adding a fallback to check the first “word” in an input containing strings against the house number I had in my database:

 for (const invitation of invitationsMatchingLastName) {
     const houseNumberMatches = invitation.houseNumber.trim().toLowerCase() === input.houseNumber.trim().toLowerCase();
-    if (houseNumberMatches) {
+    const firstWordMatches = input.houseNumber.trim().includes(" ") && input.houseNumber.trim().toUpperCase().split(" ")[0] == invitation.houseNumber.trim().toLowerCase();
+    if (houseNumberMatches || firstWordMatches) {
         await UpdateInvitation.recordLogin(ClientProvider.ddb, invitation, invitation.version, Date.now());
         await CreateAuditLog.createLog(ClientProvider.ddb, {
             logId: generateId(),

This dropped the number of failed login events to almost zero, and would not have been as easy to discover and proactively fix without these logs showing up on my dashboard.

In addition to the ability to retrieve logs by particular resource ID (i.e. logs for a particular invitation), I wanted to be able to query Dynamo for all logs in a particular timeframe (e.g. the last 48 hours). Creating a GSI using the LogDay attribute allowed doing this (without creating a hot key issue) by issuing multiple Query requests for the days included in the requested timeframe, and then filtering down further in memory:

// Get all days involved in the provided range
const daysArray: string[] = [];
for (const dt = new Date(startTimestamp); dt <= new Date(endTimestamp); dt.setDate(dt.getDate() + 1)) {
    days.push(new Date(dt).toISOString().slice(0, 10)); // Add the ISO date string up to the day
}
const logs: AuditLog[] = [];
// Iterate over all days we need to retrieve from Dynamo
for (const day of daysArray) {
    // Paginate over all logs in the day's index
    let nextToken: Record<string, string> | undefined = undefined;
        do {
            const dayLogsPage: Page<AuditLog[]> = await ReadAuditLog.readLogs(ClientProvider.ddb, day, input.resourceType, input.resourceId, nextToken);
            for (const log of dayLogsPage.items) {
                if (log.timestamp >= startTimestamp.getTime() && log.timestamp <= endTimestamp.getTime()) {
                    logs.push(log);
                }
            }
            nextToken = dayLogsPage.nextStartKey;
        } while (nextToken !== undefined)
    }

This allowed me to build an API to efficiently retrieve logs for any arbitrary timeframe, and optionally for a specific resource:

% curl -s -X POST "https://demo.wedding/api/getAuditLogs" \
  -H ... \
  -d '{"notBeforeTimestamp": "2025-12-15T00:00:00Z", "notAfterTimestamp": "2025-12-21T18:00:00Z"}' | \
  jq -r '.results[] | [.timestamp, .logType, .resourceId, .message] | @tsv'
1766255542283	UPDATE_RSVP	MJ187UHU24NP22	RSVPs were updated for 2 guests: Thomas Becker => YES, Nancy Drew => YES
1766254347071	CLIENT_ERROR	MJ187UHU24NP22	Website client encountered an error
1766254275184	LOGIN_SUCCESS	MJ187UHU24NP22	User successfully authenticated as Thomas Becker (MJ187UHU24NP22) using lastName = Becker, houseNumber = 123

Alerting

After setting up metrics, audit logs, and a canary, I needed a way to alert myself when something unexpected was happening. I signed up for PagerDuty and used its integration with CloudWatch to page me if any alarming thresholds were breached. I was really happy with PagerDuty, especially the ability to configure the paging policy to only alert me during the day and never page me overnight, and once I got all the alarm thresholds tweaked correctly, I was surprised that things were completely quiet… I should have been a bit more skeptical.

After a few weeks of letting the website run in production, I opened the PagerDuty app on my phone only to find that I had been logged out of my account. I initially assumed there was some sort of expiry on the login session, which would have been annoying, however I still would have been alerted through my backup methods (email and SMS). Instead, I tried to log back in to my account and just kept getting the same message about invalid credentials.

Having flashbacks to Square disabling my account with no warning, I went to the PagerDuty website and logged in there to finally discover the issue: my “trial” had expired.

When you sign up for a new PagerDuty account, you are offered multiple plan options ranging from free to paid. I only needed the free functionality for my use case and never used the paid plan, but despite this, it seems that any new PagerDuty account is automatically enrolled in a free trial of the paid platform. When this trial expires, not only are your logged-in applications’ credentials invalidated, but your account also stops accepting any alerts from integrations until you choose not to continue with the paid features. Assuming I must have clicked something wrong when signing up for PagerDuty originally, I restarted this process with a new account and ran into the same issue, without ever utilizing any non-free features. I tried but couldn’t even pre-empt this lockout by opting out of the paid platform early in favor of the free product.

To me, this silent lockout behavior was shocking coming from an availability platform whose sole role is alerting you when something went wrong – I would much rather just be paged to tell me that my trial had ended, and that action was required to continue coverage, than silently stop receiving alerts for an application I had already put into production. I reported this sharp edge to PagerDuty via their feedback page in September of 2025, and as of posting this it doesn’t seem to have been addressed. Although I would recommend PagerDuty once you’re fully onboarded, this is an extremely dangerous sharp edge which urgently needs fixing in my opinion.

A funny thing that happened to me as a result of having this paging setup which monitored my service running on AWS, while also working at AWS, was that I happened to be on-call for our team in EC2 during the recent DynamoDB event in us-east-1, and started receiving pages from both our internal monitoring systems and my wedding website:

Needless to say, I knew which of those alerts took priority, but I was happy at least to see that my monitoring was working correctly!!

Going Live

After working through all these details and getting the website finished, tested, and properly monitored, I was more than ready for some work on the human side of this project! Putting our RSVP vision into action was enabled using the @react-pdf/renderer library, which allowed me to generate a printable file with the individualized QR codes for each recipient. I color and font-matched these with the invitations themselves, which we designed and ordered on Minted (which we LOVED and cannot say enough good things about). I included some tiny text at the top of each RSVP card so that we could match the recipient with their envelope. The end result was a sleek experience both visually and technically.

Admin console

While eagerly awaiting the responses from our guests, I built us some protected admin pages into the website to track the progress of our responses, and to be able to export the data we and our vendors needed, such as meal choices and wedding gift details.

A view of all the invitations and who has responded

A view of each guest, their RSVP, and meal preferences

One Last Hurrah

After our incredible wedding went off without a (serious) hitch, we jumped on a plane for our honeymoon, and then returned to reality, I felt a growing hole in my heart. I really loved building this website, and I am so proud of how it turned out. Working on something so personal gives me a reason to care deeply about the small details, which is something I love to do. I came up with one last opportunity to build something tailored and fun.

I coded up a rickety but working photo-tagging interface built on Yet Another React Lightbox and React Photo Album that could be used to tag the photos we got back from our photographer to the invitations in Dynamo. I experimented with using facial recognition libraries to accelerate this process, but found that it was just unreliable enough that its result would require a manual review anyways, and it missed some people who were obvious to us as humans who remembered what people were wearing and who was together. My plan was to allow people to filter photos on the website to just the ones with them in it, so I wasn’t okay with any false-positives or misses of really good photos of them that just happened not to get auto-tagged. My wife and I put on a terrible reality TV show (Love is Blind…) got through tagging each of the 1,000+ photos before the season was over.

As a result of this work, I was able to replace our home page with a nice photo gallery experience that we sent back out to our guests so that they could relive the memories, and also have the opportunity to upload and share their own with us:

The whole photo album

A separate section for just the tagged photos of the guest

In Conclusion

Out of everything I did, there was really only one mistake in hindsight – I should have asked for people’s email addresses when RSVPing. We had their street addresses, obviously, but no way to easily communicate any reminders or updates to our guests who indicated they would be attending. Luckily this didn’t end up being too big of an issue, but if I were doing this over again I would absolutely fix that and require an email address from everyone who indicated they were coming.

I really had fun with this project and am sad that it’s now come to an end. I wrote this post as much to brag about all the work I put into this as to prolong the end of this story just a bit longer.

As silly as it may seem, this is what software engineering is all about to me – I love being able to bring just about any experience I can think of in my head to fruition, and have an impact on my life and others’. I love all the complexity and intentional decisions that sometimes underlies something that seems so simple and works well. I also am a firm believer that anything worth doing is worth overdoing, especially when it comes to something so meaningful and personal as my wedding.

And although it may seem like I spent an exorbitant amount of time on this project in the midst of such an important time for planning (and I did), I am also a believer in balance, and in sharing the load. Throughout this process my wife and I planned our wedding as a team while playing to our strengths – sometimes that involved me leaving some website ideas on the cutting room floor and making sure the actually-important things were done first. But I feel thankful that my partner saw how much joy this brought me and supported me 100% in this mostly completely unnecessary exercise in overdoing something.