Blogs

Developed a Virtual Try-On System for Make-up

Virtual Try-On technology is changing the way consumers purchase makeup. This technology combines AR and AI to create a personalized and interactive experience for consumers. They can experiment with different makeup products directly on their face, virtually and instantly. One of our clients came to us to bridge the gap between online and in-store shopping through this technology. This case study elaborates how we met our client’s expectations by enhancing user satisfaction, reducing return rates, and driving sales. 

The Problem

Online make-up shoppers were finding it difficult to choose the right products due to the lack of physical try-on options. Customers often struggled to determine the perfect shade or match for their skin tone. This led to low confidence and higher return rates among consumers. 

The Goal

To create a virtual solution that allows customers to experiment with different shades of lipstick, eyeshadow, foundation, and concealer without physically applying products. This would help them find their ideal look.

The Tech Stack

Python

We utilized Python as the main programming language due to its simplicity and wide range of libraries for computer vision, image processing, and AI.

OpenCV

We used OpenCV to read, process, and manipulate facial images. It was responsible for applying makeup colors to different facial features like cheeks, lips, eyes, and hair. Image processing techniques like Gaussian blur and morphological operations helped improve the realism of the makeup.

Facial Landmarks Detection

We made the most of Libraries like Dlib and MediaPipe to detect facial landmarks. They helped identify facial regions such as eyelids, eyes, lips, cheeks, forehead, and hair, which are essential for targeted makeup application.

Morphological Image Processing

We applied Morphological operations like erosion, dilation, opening, and closing to refine the makeup regions and improve blending.

Generative-AI

We trained a Generative-AI model that was used for face reconstruction based on Vision Transformers. It was used to enhance the final makeup image by making the colors blend naturally with the skin and producing a more realistic and photo-like result.

The Process

The process starts once the user uploads their face image to the application. The uploaded image is first passed to facial landmark detection models like Dlib or MediaPipe. These models identify important facial regions such as the eyelids, eyes, lips, forehead, cheeks, head, and hair. 

Once these facial features are detected, the system identifies the areas where makeup will be applied—for example, lipstick to lips, eyeshadow to eyelids, eyeliner to eyes, blush or base to cheeks, and optional hair coloring in the hair region. 

After the regions are identified, OpenCV is used to apply makeup by processing the image. Colors are blended into the facial regions. Initially, the makeup may appear artificial, as if it’s painted on the image. To address this, image processing techniques are applied.

Gaussian blur is used to soften the edges and improve blending. Morphological Image processing is used to refine the applied makeup. It gives natural-looking results by cleaning up the regions and improving how the makeup blends with the skin.

Also Read: https://nextbridge.com/developing-a-rag-system-for-scalable-enterprise-search/

Despite these improvements, the makeup can still look unnatural or too digital. To overcome this, the final image after makeup application and image processing is passed through a deep learning model. This model is based on a vision transformer used for high-quality face reconstruction. It enhances textures, smooths out transitions, and gives a more photo-realistic finish, making the makeup appear as if it’s naturally applied.

Finally, the output image, now featuring natural-looking virtual makeup, is displayed back to the user. They can view the results, try different shades, or save the image for later

The results

We successfully delivered the following benefits to our client through the Virtual Try-On System:

1. Enhanced User Experience: 

Virtual try-on offered customers an engaging and interactive way to explore and test makeup products from the comfort of their homes. It replicated the in-store try-on experience, making online shopping more enjoyable and informative.

2. Increased Sales: 

By allowing customers to see how makeup products looked on their faces, we reduced purchase hesitancy and encouraged more sales.

3. Reduced Return Rates:

Virtual try-on helped customers make more informed choices, resulting in fewer purchases of products that did not suit their skin tone or preferences. This led to a decrease in return rates and associated costs.

4. Personalization: 

The technology can analyze a user's skin tone and facial features to recommend makeup shades and products that match their characteristics. This personalization boosts customer satisfaction and loyalty.

5. User-Generated Content: 

Virtual try-on encouraged users to create and share content on social media, showcasing the products they virtually tried on. This user-generated content acted as free advertising and generated buzz around the brand.

6. Competitive Advantage: 

Implementing virtual try-on put eCommerce store ahead of competitors who were not offering this feature. It demonstrated their commitment to providing cutting-edgeand customer-centric solutions.

7. Data Insights: 

The technology collected valuable data on user preferences, allowing the client to make data-driven decisions about which products to feature and promote.

8. Cross-Sell and Upsell Opportunities: 

Virtual try-on suggested complementary products or upsell higher-end items based on the user's chosen makeup look, increasing the average order value.

9. Customer Engagement: 

Virtual try-on did not just act like a shopping tool; it was also a form of entertainment. Users spent more time on the client’s site, increasing overall engagement.

Next Steps

We plan to extend the virtual try-on makeup to intelligent facial makeup recommendations for the following categories:

• Virtual Try-On glasses and jewelry

• Virtual Try-On digital gadgets like watches

• Virtual Try-On clothes

• Virtual Try-On shoes

Want to explore various fashion and beauty options without the need to physically put them on for your customers? Connect with us.