So I've found, no need of expensive ATmel ICE for programming ATMega4809 UPDI program anymore!!

Eltangus already implemented UPDI programmer firmware also compatible with Arduino at GitHub, which indeed makes ATMega328P working as a UPDI speaking STK500!! The procedure to make it is quite easy.

(Step 1) Write UPDI firmware on your Arduino UNO and wiring with ATMega4809 UPDI

The pre-compiled binary is available at Eltangus's github. You may need Atmel-ICE (or directly from Arduino IDE) and after writing it, your UNO will turns into UPDI programmer. Please don't forget "disable auto RESET" by disabling capacitor connecting from DTR of USB-UART to RST of Mega328P.

As is written in Eltangus's page, wiring is very simple. Just connecting PD6 (D6 of Arduino pin) and UPDI of ATMega4809 with VCC and GND is enough to complete.

(Step 2) Setting up on ATmel Studio

Original state of Atmel Studio will not recognize the existence of "UPDI-STK500 by UNO". Just click "Add target" and

Select tool as "STK500" and select appropriate serial port on your system. Now almost done!

Now the time for programming ATmega4809 through UPDI. STK500 does not know ATmega4809 and just select "ATmega644P" and check off "Erase device before programming" and click "Program".... (Don't need to think too serious, for example interface selection. Just default ISP is enough to complete). As for the implementation, some function is not implemented yet but at least we can write binary through UPDI.

You will see the binary is uploaded to ATmega4809 !!!