[api][filebrowser] Complete API redesign with comprehensive REST endpoints, validation, and separation of concerns

[Harshg999]

What changes were proposed in this pull request?

• Major architectural refactoring of the Hue filebrowser introducing a modern, comprehensive REST API with enhanced validation, error handling, and multi-filesystem support.

Key Changes

API Architecture:

• 17 new REST endpoints with resource-based design (/api/v1/storage/*)
• FileAPI, DirectoryAPI with CRUD operations
• Comprehensive operation endpoints (exists, copy, move, delete, permissions etc)

Validation Framework:

• Pydantic schemas with security-first validation
• DRF serializers wrapping Pydantic for multi-layer validation
• Path traversal protection, file extension restrictions, size limits etc

Business Logic Redesign:

• Pure, filesystem-agnostic operations in operations.py
• Strategy Pattern file readers (gzip, bz2, snappy, avro, parquet)
• Atomic operations with enhanced error handling

Multi-Filesystem Support:

• Enhanced ProxyFS for seamless HDFS/S3/Azure/GCS operations
• Unified abstraction with consistent behavior across platforms

Technical Improvements:

• Range request support for file downloads and video streaming
• Streaming operations for memory efficiency
• Comprehensive logging and structured error responses

File Changes:

• api.py: Complete REST API implementation
• operations.py: Pure business logic functions
• schemas.py: Comprehensive Pydantic validation
• serializers.py: DRF integration layer
• utils.py: Enhanced utilities with FileReader classes
• s3fs.py/abfs.py/gs.py/ofs.py: Cloud storage optimizations
• api_public_urls_v1.py: New endpoint routing

How was this patch tested?

• WIP

[github-actions]

⚠️ No test files modified. Please ensure that changes are properly tested. ⚠️

[github-actions]

UI Code Coverage Report

Lines	Statements	Branches	Functions
	39.56% (30941/78205)	31.34% (14450/46101)	25.01% (2251/8998)

[github-actions]

Backend Code Coverage Report •

File	Stmts	Miss	Cover	Missing
apps/filebrowser/src/filebrowser
api.py	509	428	15%	113–120, 128, 130–132, 134–135, 137, 147–151, 170–172, 174–176, 206, 208–213, 216–218, 221–225, 227–230, 244–246, 248–252, 273–275, 277–279, 281–284, 286–287, 290, 292, 294, 296–299, 301–302, 304, 307–308, 310–312, 314–315, 317–322, 325, 333–334, 336, 338–340, 344–345, 347, 349–354, 372, 374–375, 377–380, 382–386, 398, 400–403, 405–407, 409–411, 413–415, 417, 419–421, 423–425, 427–428, 431–434, 437–439, 442–443, 446–447, 449–452, 454–455, 457–460, 463–465, 468–471, 473, 478–479, 481–486, 490–492, 494–496, 498–502, 505–506, 508–511, 520–522, 524–526, 528–542, 551–553, 555–557, 559–573, 591–593, 595–597, 599–613, 622–624, 626–628, 630–647, 655–657, 659–661, 663–667, 675, 677–678, 680–682, 684–688, 692–694, 696–700, 706–708, 710–712, 714–718, 724–726, 728–730, 732–736, 742–744, 746–748, 750–754, 760–762, 764–766, 768–772, 778–780, 782–784, 786–790, 796–798, 800–802, 804–808, 814–816, 818–820, 822–826, 832–834, 836–838, 840–844, 850–852, 854–856, 858–862, 868–870, 872–874, 876–880, 886–888, 890–894
operations.py	430	393	8%	81–82, 84–85, 87, 90–92, 95, 98–99, 102–103, 105, 107–111, 113, 135–137, 139, 141, 143, 145, 149–155, 157, 186–188, 190, 193–195, 198–199, 201, 203–209, 211–212, 214–215, 217–219, 242–244, 246, 248–251, 254–255, 257–259, 263, 265–271, 273–275, 277, 279–280, 282–284, 308, 310–311, 313, 316–317, 319–320, 323–326, 329–336, 338, 340, 342, 344–346, 349, 351, 353–356, 358–360, 362–365, 367, 370–374, 377–380, 382, 394–395, 397–399, 423, 425–426, 428, 431–433, 435, 437–441, 444–445, 447, 452, 454–455, 457, 459–460, 462–463, 466–469, 471–472, 474–477, 479–480, 482, 487, 489–490, 492–493, 495–498, 500–501, 503, 508, 510–511, 513, 515–516, 518, 520–522, 524–527, 529, 531–532, 534–535, 537, 543–544, 546–547, 549–550, 552–554, 559, 561–562, 564, 566–567, 569, 571–573, 575–576, 578–579, 581, 586, 588–589, 591–592, 594–597, 599–600, 602–603, 605, 610, 612–613, 615–616, 618–621, 623–624, 626, 631, 633–634, 636–637, 639, 652–655, 657–658, 660–661, 663, 668, 670–671, 673–674, 676–679, 681–682, 684–685, 687, 692, 694–695, 697, 699–700, 702–707, 709–710, 712, 717, 719–720, 722–723, 725–726, 729–732, 734, 736–740, 745, 747–748, 750–751, 753–754, 757–760, 762, 764–768, 773, 775–776, 778, 780–782, 784–787, 789, 791–794, 799, 801–802, 804, 806–811, 816, 818–819, 821–825, 827, 829, 842, 844–845, 847–848, 850–854, 856–857, 859–860, 862–863
schemas.py	374	192	48%	38–40, 45–47, 52–54, 59–61, 64–67, 69, 89–91, 95–96, 98–99, 101–105, 107–108, 111–112, 115–116, 118, 140–141, 143, 145–146, 149–150, 153–154, 156, 162–166, 172, 174–177, 181, 184–186, 188, 209–210, 213, 216–217, 220–221, 224–225, 227, 233–237, 243, 245–248, 254, 256–257, 260–263, 265, 289–290, 293, 296–297, 300–301, 303, 309–312, 318–321, 324–328, 334–336, 338, 357–358, 361, 364–365, 368–369, 371, 383–385, 396–401, 413–418, 423–425, 437–443, 445–447, 459–464, 475–480, 503–508, 522–527, 531–533, 545–547, 560–562, 567–569, 581–583, 594–596
serializers.py	201	95	52%	60–63, 65, 85–89, 105–106, 108–109, 111, 125–129, 143–147, 167–171, 182–186, 198–202, 211–215, 225–229, 239–243, 253–257, 266–270, 292–296, 308–312, 322–326, 337–341, 351–355, 364–368
utils.py	406	328	19%	48–49, 55–56, 61, 95–96, 98–99, 101, 105–114, 118–123, 125–129, 131–135, 137–140, 143–145, 149–153, 157–165, 167, 171–180, 182–185, 187, 191–200, 202, 217–218, 220–222, 225–226, 228–230, 233–234, 236–238, 240, 256–257, 264, 295, 310–312, 315, 317–324, 326–327, 329, 331–333, 341–342, 344–349, 357, 359–363, 371–372, 374–376, 378–380, 383–385, 387–389, 397–398, 400–402, 404–406, 409–412, 415–416, 418–419, 422, 424–426, 434–435, 437, 439, 441–442, 445, 448, 450, 452–454, 470, 479–481, 483, 488, 493, 498, 508–509, 511–515, 520–527, 543–546, 549–559, 561–565, 567, 595, 597–600, 603–606, 609, 612–613, 616, 618, 620, 622–624, 627–631, 653, 655–656, 658, 660–662, 665, 668–671, 673, 675–676, 679–684, 686, 696–697, 700–701, 704–705, 707, 709–711, 714–716, 718–720, 722, 724–726, 728, 730, 735, 790, 793–798, 801–806, 809–814, 817–818, 820, 835, 843–844, 846, 859, 861–862, 864–867, 876
desktop/core/src/desktop
api_public.py	266	140	47%	46–47, 52–53, 59, 64–65, 70–71, 76–77, 82–83, 88–89, 97–98, 103–104, 109–110, 115, 117–119, 121, 129, 136–139, 141, 143–144, 146, 151, 153, 155, 160, 162, 164, 169–170, 175–176, 181–182, 187–188, 193, 195, 197, 202, 204, 206, 211, 213, 215, 220–221, 226–227, 232, 234–235, 237, 245–246, 251–252, 260–261, 269–270, 275–276, 281–282, 290–291, 296–297, 302–303, 308–309, 314–315, 320–321, 326–327, 332–333, 341–342, 347–348, 353–354, 359–360, 365–366, 371–372, 377–378, 383–384, 389–390, 395–396, 401–402, 410–411, 416–417, 422–423, 430–433, 438, 440, 444, 446–447, 449–450, 454, 456, 459–460, 462–463, 465
desktop/core/src/desktop/lib/fs
proxyfs.py	205	135	34%	38–39, 43–47, 50, 53–54, 56, 59–66, 69–72, 74–77, 81–83, 86–93, 95–97, 105–109, 113–115, 117–118, 121, 124–127, 129, 132–135, 140, 143, 146, 149, 152, 155, 158, 161, 164, 167, 170, 173, 176, 179, 182, 185, 188, 191, 194, 197, 200, 203, 206, 213, 215–218, 221–222, 225–230, 235–236, 238–240, 243, 246, 249, 252–253, 256–258, 263–265, 268, 271–273, 276, 279–281, 284, 287–289, 292, 295–297, 300, 304, 307, 310, 313, 316
desktop/core/src/desktop/lib/fs/gc
gs.py	288	217	24%	43, 48, 53–64, 66–71, 76, 82–85, 87, 89, 91, 97, 107, 111, 114, 126, 128–129, 131–132, 134–136, 138, 142–147, 149, 164–168, 183–187, 189–191, 193–194, 196, 199–200, 202–208, 210–211, 213–215, 219–222, 237–238, 240, 242–243, 245–249, 252–254, 256–258, 260–263, 265–267, 269, 273, 288–289, 291–293, 295, 297–298, 300–304, 306, 308–310, 314–315, 324, 326–327, 329–336, 338–341, 343, 345–346, 349–350, 352–360, 362–364, 366–368, 370, 372, 376–377, 379, 381–382, 384–385, 387, 404–406, 409–412, 415–416, 418–419, 421–423, 427–430, 432–434, 436–438, 440, 443–444, 446, 460, 463–465, 467, 473–474, 476, 479–480, 483, 486, 489, 492, 495, 498, 501, 504
desktop/core/src/desktop/lib/fs/ozone
ofs.py	217	173	20%	45, 54, 64–66, 68–70, 72, 76, 88, 91–94, 96, 99, 102, 105, 108, 116–117, 119–120, 122–125, 127, 129–130, 136–138, 140–143, 145–150, 152, 155, 173, 179–182, 203, 205–207, 209–210, 212–214, 216–217, 220, 223–228, 230, 235–236, 239–241, 244, 247, 249–250, 252, 255, 258–261, 263, 265–268, 270, 274–276, 278–279, 281–284, 286–287, 290, 297, 301–304, 306–307, 309–310, 312, 314–315, 319–320, 322–325, 327–329, 332–333, 335–336, 338–339, 341–345, 347–349, 351–352, 371–372, 377–379, 381–382, 384–385, 387, 389, 392–394, 399–401, 403, 405, 409, 414–417, 419, 421, 423, 426, 428, 431, 434, 437, 440
desktop/libs/aws/src/aws/s3
s3fs.py	483	359	25%	47, 52, 57–68, 70–75, 80, 86–88, 93, 95, 97, 99, 104–112, 115–119, 123, 126, 128–132, 135–141, 143–147, 149–150, 158–159, 161, 164, 166, 168–174, 177, 180–181, 183–189, 191–192, 194, 197–198, 200, 203–204, 206–214, 216–221, 225–227, 229–233, 237–239, 243, 247–248, 252, 256, 259, 263–267, 269–272, 276–279, 283–285, 289–292, 296–299, 303, 307–311, 315–316, 318–320, 322–326, 329–331, 333–339, 341–344, 348, 369–370, 372–375, 377–378, 380–381, 384–386, 388, 390–393, 395–396, 398–399, 401–403, 406–408, 411, 413, 416–417, 420–421, 423–429, 431–432, 435–439, 441, 446, 449, 452, 456–461, 463, 473–475, 477–484, 486–489, 491–493, 498, 503–505, 510, 513–515, 517–520, 523–524, 526–527, 529–531, 533–536, 538–540, 546–551, 553–554, 556, 558–560, 565, 568–570, 572, 578–579, 581, 586–592, 597–598, 603–604, 606–609, 611–613, 615–616, 618–619, 621–623, 625–626, 631, 636–639, 643–649, 651–655, 658, 661–662, 665–666, 669, 672, 675, 678, 681, 684, 687
desktop/libs/azure/src/azure/abfs
abfs.py	460	368	20%	48, 68–87, 90–91, 93, 97–98, 114–115, 117, 119, 122, 126–127, 129, 133, 137, 143–146, 148–150, 154–157, 159–161, 167–177, 184–189, 191–192, 194, 201–202, 204–215, 217–223, 226–227, 229, 231, 237–240, 242–243, 245–246, 248, 255–257, 259, 261, 268–269, 272–273, 275, 277, 279, 285–286, 288, 290, 296–297, 304, 313, 320–321, 328, 335, 342, 350–352, 354, 360, 362, 364–365, 369–373, 375–377, 379, 385–393, 395, 401, 409–412, 414, 420, 425–428, 430, 436, 438–441, 443, 445–446, 448–451, 453, 455, 461–471, 473, 482, 488, 494–495, 497–501, 503–506, 508, 511, 519–523, 525, 531–534, 536, 538, 544–547, 549, 552, 555, 563–565, 571–575, 581–586, 592–593, 595–600, 602, 608, 615, 621–622, 624–625, 627–628, 634, 636–638, 640–641, 643, 649–652, 654–668, 670, 672, 675, 678, 681, 684, 705, 707–709, 711–712, 714–716, 718–719, 721, 724–729, 731, 736–737, 740–742, 745, 748, 750–751, 753, 756, 759–762, 764, 766–769, 771, 775–777, 779–780, 782–785, 787–788, 794–796, 798–799, 805, 811, 814–815, 818, 837, 839, 841–843, 845, 848, 850–851, 854–855, 858, 866–869
TOTAL	55499	41327	25%

[bjornalm]

Amazing work, I'm gonna let the backend developers look into the code more closely but I have a few questions.

• Should we also try to break up the 900 lines large api.py file into smaller more manageable files by grouping functionality?
• Will these APIs also suport the old file browser (seeing deprecated code)?
• If we are breaking the public API we should update the API version right?
• What about the unit tests (we want unit tests for all PR)?
• Do we wanna keep the LOG.info prints?
• Should we take this opportunity and introduce the improved error format?

[bjornalm]

Are we gonna implement the new Error structure you (and I) designed?

[ranade1]

please provide tests for this big change.