Shulou(Shulou.com)05/31 Report--

Today, I would like to share with you the relevant knowledge about what the Linux dynamic tracking tool is. The content is detailed and the logic is clear. I believe most people still know too much about this, so share this article for your reference. I hope you can get something after reading this article. Let's take a look at it.

Thread and memory profiling can only observe the overall situation of the process. Sometimes we need to observe a certain method level, such as what are the parameters passed in when calling the method test (), what is the return value, and how long does it take? In this case, we need to use some dynamic tracking tools, such as strace, arthas, bpftrace, systemtap and so on.

Strace and ltrace

Strace is a tool used to observe system calls in Linux. As we all know, the operating system exposes a number of system call interfaces to applications, and applications can only access the operating system through these system call interfaces, such as applying for memory, files or network io operations.

The usage is as follows:

#-T print time spent on system call #-tt prints the maximum length of system call #-s output. Default is 32. For scenarios with long call parameters, it is recommended to increase whether #-f tracks the system calls of child processes from fork. Since thread pools are commonly used by server services, it is recommended to add #-p to specify the tracking process pid#-o to which file the trace log is output. If not specified, it is directly output to the terminal $strace-T-tt-f-s 10000-p 87-o strace.log

Example: grab the SQL actually sent

Sometimes, we will find that there is no problem with the SQL in the code, but the data can not be found, which is most likely due to the fact that some of the underlying frameworks in the project have rewritten the SQL, resulting in a difference between the actual sent SQL and the SQL in the code.

In this case, do not rush to pick the underlying framework code, that will take more time, after all, the programmer's time is very precious, otherwise you have to work more shifts, how to quickly confirm whether this is the reason?

There are two methods. The first is to use wireshark to grab packets, and the second is the strace introduced in this article. Because the program must send SQL commands to the database through network io-related system calls, we only need to use strace to track all system calls, and then grep the system calls that send SQL, as follows:

$strace-T-tt-f-s 10000-p 87 | & tee strace.log

You can see clearly from the figure that the jdbc driver of mysql sends SQL through sendto system calls, and obtains the return result through recvfrom. It can be found that because SQL is a string, strace automatically identifies it for us, while the return result is not easy to identify because it is binary, so you need to be very familiar with mysql protocol.

In addition, from the above, it is easy to see that the execution of SQL is time-consuming, so you can calculate the time difference between sendto and recvfrom with the same thread number.

Ltrace

Since most processes basically use the underlying c library instead of system calls, such as msvc on glibc,Windows on Linux, there is also a tool ltrace that can be used to trace library calls, as follows:

$ltrace-T-tt-f-s 10000-p 87-o ltrace.log

The basic usage is the same as strace. Generally speaking, using strace is enough.

Arthas

Arthas is a dynamic tracking tool under java, which can observe the call parameters and return values of java methods. In addition, it also provides many practical functions, such as decompilation, thread analysis, heap memory dump, flame graph and so on.

Download and use

# download arthas$ wget https://arthas.aliyun.com/download/3.4.6?mirror=aliyun-O arthas-packaging-3.4.6-bin.zip# extract $unzip arthas-packaging-3.4.6-bin.zip-d arthas & & cd arthas/# enter the arthas command interface $java-jar arthas-boot.jar `java` [INFO] arthas-boot version: 3.4.6 [INFO] arthas home: / home/work/arthas [INFO] Try to attach process 3368243 [INFO] Attach process 3368243 success. [INFO] arthas-client connect 127.0.0.1 3658 -. . . ,. ,. ,. / O\ |. .--'|'|'-'| / O\'. -'|. | |'-. |. -. | |.. | `. | `-. |\ |. -'|`--'`- -'--'wiki https://arthas.aliyun.com/doctutorials https://arthas.aliyun.com/doc/arthas-tutorials.htmlversion 3.4.6pid 3368243time 2021-- 11-13 13 arthas@3368243 35 help# help watch help to see which commands are provided by arthas [arthas@3368243] $help# help watch to view the specific usage of the watch command [arthas@3368243] $help watch

Watch, trace and stack

In arthas, you can observe the method invocation using the watch, trace, stack commands, as follows:

# query executed by watch observation SQL,-x 3 specifies the object deployment level [arthas@3368243] $watch org.apache.ibatis.executor.statement.PreparedStatementHandler parameterize'{target.boundSql.sql,target.boundSql.parameterObject}'- x 3method=org.apache.ibatis.executor.statement.PreparedStatementHandler.parameterize location=AtExitts=2021-11-13 14:50:34 [cost=0.071342ms] result=@ArrayList [@ string [select id,log_info,create_time,update_time,add_time from app_log where id=?], @ ParamMap [@ string [id]: @ Long [41115], @ string [param1]: @ Long [41115],] ] # watch observation time-consuming SQL [arthas@3368243] $watch com.mysql.jdbc.PreparedStatement execute'{target.toString ()} 'target.originalSql.contains ("select") & & # cost > 200'-x 2Press Q or Ctrl+C to abort.Affect (class count: 3, method count: 1) cost in 123 ms, listenerId: 25method=com.mysql.jdbc.PreparedStatement.execute location=AtExitts=2021-11-13 14:58:42 [cost=1001.558851ms] result=@ArrayList [@ string [com.mysql.jdbc.PreparedStatement @ 6283cfe6: select count (*) from app_log],] # trace tracking method is time-consuming, and the root cause of time-consuming can be found by tracking layer by layer.-- skipJDKMethod false shows that the jdk method is time-consuming. [arthas@3368243] $trace com.mysql.jdbc.PreparedStatement execute 'target.originalSql.contains ("select") & & # cost > 200'-skipJDKMethod falsePress Q or Ctrl+C to abort.Affect (class count: 3, method count: 1) cost in 191 ms is not displayed by default, listenerId: 26---ts=2021-11-13 15:00:40 Thread_name=http-nio-8080-exec-47;id=76;is_daemon=true;priority=5 TCCL=org.springframework.boot.web.embedded.tomcat.TomcatEmbeddedWebappClassLoader@5a2d131d-[1001.465544ms] com.mysql.jdbc.PreparedStatement:execute () +-- [0.022119ms] com.mysql.jdbc.PreparedStatement:checkClosed () # 1274 +-[0.016294ms] com.mysql.jdbc.MySQLConnection:getConnectionMutex () # 57 +-[0.017862ms] com.mysql.jdbc.PreparedStatement:checkReadOnlySafeStatement () # 1278 +- -- [0.008996ms] com.mysql.jdbc.PreparedStatement:createStreamingResultSet () # 1294 +-- [0.010783ms] com.mysql.jdbc.PreparedStatement:clearWarnings () # 1296 +-- [0.017843ms] com.mysql.jdbc.PreparedStatement:fillSendPacket () # 1316 +-[0.008543ms] com.mysql.jdbc.MySQLConnection:getCatalog () # 1320 +-- [0.009293ms] java.lang.String:equals () # 57 +-[0.008824ms] com.mysql.jdbc.MySQLConnection:getCacheResultSetMetadata () # 1328 +-- [0.009892ms] com.mysql.jdbc.MySQLConnection:useMaxRows () # 1354 +-[1001.055229ms] com.mysql.jdbc.PreparedStatement:executeInternal () # 1379 +-- [0.02076ms] com.mysql.jdbc.ResultSetInternalMethods:reallyResult () # 1388 +-- [0.011517ms] com .mysql.jdbc.MySQLConnection: getCacheResultSetMetadata () # 57 +-- [0.00842ms] com.mysql.jdbc.ResultSetInternalMethods:getUpdateID () # 1404-[0.008112ms] com.mysql.jdbc.ResultSetInternalMethods:reallyResult () # 140 trace stack trace method call stack Find the time-consuming SQL source [arthas@3368243] $stack com.mysql.jdbc.PreparedStatement execute 'target.originalSql.contains ("select") & & # cost > 200'Press Q or Ctrl+C to abort.Affect (class count: 3, method count: 1) cost in 138 ms, listenerId: 27ts=2021-11-13 15:01:55 Thread_name=http-nio-8080-exec-5;id=2d;is_daemon=true;priority=5 TCCL=org.springframework.boot.web.embedded.tomcat.TomcatEmbeddedWebappClassLoader@5a2d131d @ com.mysql.jdbc.PreparedStatement.execute () at com.alibaba.druid.pool.DruidPooledPreparedStatement.execute (DruidPooledPreparedStatement.java:493) at org.apache.ibatis.executor.statement.PreparedStatementHandler.query (PreparedStatementHandler.java:63) at org.apache.ibatis.executor.statement.RoutingStatementHandler.query (RoutingStatementHandler.java:79) at org.apache.ibatis.executor.SimpleExecutor.doQuery (SimpleExecutor) . Java: 63) at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase (BaseExecutor.java:326) at org.apache.ibatis.executor.BaseExecutor.query (BaseExecutor.java:156) at org.apache.ibatis.executor.BaseExecutor.query (BaseExecutor.java:136) at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList (DefaultSqlSession.java:148) at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList (DefaultSqlSession. Java:141) at org.apache.ibatis.session.defaults.DefaultSqlSession.selectOne (DefaultSqlSession.java:77) at sun.reflect.GeneratedMethodAccessor75.invoke (null:-1) at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke (Method.java:498) at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke (SqlSessionTemplate.java:433) at com.sun.proxy.$Proxy113 .selectOne (null:-1) at org.mybatis.spring.SqlSessionTemplate.selectOne (SqlSessionTemplate.java:166) at org.apache.ibatis.binding.MapperMethod.execute (MapperMethod.java:83) at org.apache.ibatis.binding.MapperProxy.invoke (MapperProxy.java:59) at com.sun.proxy.$Proxy119.selectCost (null:-1) at com.demo.example.web.controller.TestController.select (TestController.java:57)

You can see that conditional expressions can be specified in watch, trace, and stack commands. As long as the syntax of ognl expression is satisfied, the complete syntax of ognl is very complex. Here are some frequently used ones:

Ognl

With the ognl command, you can view the values of static variables directly, as follows:

# call the System.getProperty static function to look at the jvm default character encoding [arthas@3368243] $ognl'@ System@getProperty ("file.encoding")'@ string [UTF-8] # find the springboot class loader [arthas@3368243] $classloader-t+-BootstrapClassLoader+-sun.misc.Launcher$ExtClassLoader@492691d7 +-sun.misc.Launcher$AppClassLoader@764c12b6 +-org.springframework.boot.loader.LaunchedURLClassLoader@4361bd48# to get the classloader hash values of all beanName,-c specified springboot in springboot # General Spring project Will define a SpringUtil. Used to get the bean container ApplicationContext [arthas@3368243] $ognl-c 4361bd48'# context=@com.demo.example.web.SpringUtil@applicationContext, # context.beanFactory.beanDefinitionNames'@String [] [@ string [org.springframework.context.annotation.roomConfigurationAnnotationProcessor], @ string [org.springframework.context.annotation.originAutowiredAnnotationProcessor], @ string [org.springframework.context.annotation.roomCommonAnnotationProcessor], @ string [testController], @ string [apiController], @ string [loginService] ...] # get springboot configuration For example, server.port is [arthas@3368243] $ognl-c 4361bd48'# context=@com.demo.example.web.SpringUtil@applicationContext, # context.getEnvironment () .getProperty ("server.port")'@ string [8080] # to see which configuration file server.port is defined in # it's easy to see Server.port is defined in application-web.yml [arthas@3368243] $ognl-c 4361bd48'# context=@com.demo.example.web.SpringUtil@applicationContext, # context.environment.propertySources.propertySourceList. {? ContainsProperty ("server.port")}'@ ArrayList [@ ConfigurationPropertySourcesPropertySource [ConfigurationPropertySourcesPropertySource {name='configurationProperties'}], @ OriginTrackedMapPropertySource [OriginTrackedMapPropertySource {name='applicationConfig: [classpath:/application-web.yml]'}],] # call the bean method in springboot Get the return value [arthas@3368243] $ognl-c 4361bd48'# context=@com.demo.example.web.SpringUtil@applicationContext, # context.getBean ("demoMapper"). QueryOne (12)'- x 2@ArrayList [@ HashMap [@ string [update _ time]: @ Timestamp [2021-11-09 1838 time 13000], @ string [create _ time]: @ Timestamp [2021-04-17 152context=@com.demo.example.web.SpringUtil@applicationContext 55000], @ string [info]: @ string [TbTRNsh3SixuFrkYLTeb25a6zklEZj0uWANKRMe] @ string [id]: @ Long [12], @ string [add _ time]: @ Integer [61],],] # View springboot's thread pool with tomcat [arthas@3368243] $ognl-c 4361bd48'# context=@com.demo.example.web.SpringUtil@applicationContext, # context.webServer.tomcat.server.services [0] .connectors [0] .roomHandler.endpoint.executor'@ ThreadPoolExecutor [sm=@StringManager [org.apache.tomcat.util.res.StringManager@16886f49] SubmittedCount=@AtomicInteger [1], threadRenewalDelay=@Long [1000], workQueue=@TaskQueue [isEmpty=true Size=0], mainLock=@ ReentrantLock [java.util.concurrent.locks.ReentrantLock @ 69e9cf90 [unloaded]], workers=@HashSet [isEmpty=false Size=10], largestPoolSize=@Integer [49], completedTaskCount=@Long [10176], threadFactory=@TaskThreadFactory [org.apache.tomcat.util.threads.TaskThreadFactory@63c03c4f], handler=@RejectHandler [org.apache.tomcat.util.threads.ThreadPoolExecutor$RejectHandler@3667e559], keepAliveTime=@Long [60000000000], allowCoreThreadTimeOut=@Boolean [false], corePoolSize=@Integer [10], maximumPoolSize=@Integer [8000],]

Other commands

Arthas also provides jvm disk, thread profiling, heap dump, decompilation, flame diagram and other functions, as follows:

# shows the first four threads that consume more cpu [arthas@3368243] $thread-n 4 "C2 CompilerThread0" [Internal] cpuUsage=8.13% deltaTime=16ms time=46159ms "C2 CompilerThread1" [Internal] cpuUsage=4.2% deltaTime=8ms time=47311ms "C1 CompilerThread2" [Internal] cpuUsage=3.06% deltaTime=6ms time=17402ms "http-nio-8080-exec-40" Id=111 cpuUsage=1.29% deltaTime=2ms time=624ms RUNNABLE (in native) at java.net.SocketInputStream.socketRead0 (Native Method). At com.mysql.jdbc.MysqlIO.checkErrorPacket (MysqlIO.java:4113) at com.mysql.jdbc.MysqlIO.sendCommand (MysqlIO.java:2570) at com.mysql.jdbc.MysqlIO.sqlQueryDirect (MysqlIO.java:2731) at com.mysql.jdbc.ConnectionImpl.execSQL (ConnectionImpl.java:2818). At com.demo.example.web.controller.TestController.select (TestController.java:57) # heap dump [arthas@3368243] $heapdumpDumping heap to / tmp/heapdump2021-11-13-15-117226383240040009563.hprof... Heap dump file created# cpu flame diagram, profiler start may not be available in container environment You can use profiler start-e itimer instead of [arthas@3368243] $profiler startStarted [cpu] profiling [arthas@3368243] $profiler stopOKprofiler output file: / home/work/app/arthas-output/20211113-151208.svg# dashboard, just like top under Linux, you can see the overall situation of jvm threads and heap memory [arthas@3368243] $dashboard# jvm is similar to ps under Linux, you can see some basic information of jvm process. For example, jvm parameter, class load, number of threads, number of open file descriptors, etc. [arthas@3368243] $jvm# decompiled [arthas@3368243] $jad com.demo.example.web.controller.TestController

It can be seen that arthas is no longer a simple dynamic tracking tool, it includes almost all the diagnostic functions commonly used under jvm.

Bpftrace

Arthas can only track java programs, but there is nothing you can do about native programs (such as MySQL). Fortunately, Linux Ecology provides a large number of mechanisms and supporting tools, which can be used to track the calls of native programs, such as perf, bpftrace, systemtap and so on. Because it is not difficult to use bpftrace, this article mainly introduces its usage.

Bpftrace is a dynamic tracking tool based on ebpf technology. It encapsulates ebpf technology and implements a scripting language, just like the arthas based on ognl described above. It implements a scripting language similar to awk, encapsulates common statement blocks, and provides built-in variables and built-in functions, as follows:

$sudo bpftrace-e 'BEGIN {printf ("Hello, World!\ n");}' Attaching 1 probe...Hello, World!

Example: tracking slow SQL on the calling side

Earlier, we used strace to track the jdbc driver of mysql, which uses sendto and recvfrom system calls to communicate with the mysql server. Therefore, when we call sendto, we count the time point, and then at the end of recvfrom, we calculate the time difference, and we can get the corresponding SQL time, as follows:

First find the tracking points of sendto and recvfrom system calls in bpftrace, as follows:

# find the trace point of sendto | recvfrom system call. You can see that the trace point at the beginning of sys_enter_ should be triggered when entering, and $sudo bpftrace-l'* tracepoint:syscalls*' is triggered when exiting at the beginning of sys_exit_ | grep-E 'sendto | recvfrom'tracepoint:syscalls:sys_enter_sendto tracepoint:syscalls:sys_exit_sendto tracepoint:syscalls:sys_enter_recvfrom tracepoint:syscalls:sys_exit_recvfrom # View system call parameters It is convenient for us to write a script $sudo bpftrace-lv tracepoint:syscalls:sys_enter_sendtotracepoint:syscalls:sys_enter_sendto int _ _ syscall_nr Int fd; void * buff; size_t len; unsigned int flags; struct sockaddr * addr; int addr_len

Write a trace script trace_slowsql_from_syscall.bt with the following code:

#! / usr/local/bin/bpftraceBEGIN {printf ("Tracing jdbc SQL slower than% d ms by sendto/recvfrom syscall\ n", $1); printf ("%-10s%-6s% 6s% s", "TIME (ms)", "PID", "MS", "QUERY") } tracepoint:syscalls:sys_enter_sendto / comm = = "java" / {/ / mysql protocol, the fifth byte at the beginning of the packet indicates the command type, 3 represents the SQL query $com = * (uint8 *) args- > buff) + 4); if ($com = = (uint8) 3) {@ query[ tid] = str (uint8 *) args- > buff) + 5, (args- > len)-5); @ start [tid] = nsecs }} tracepoint:syscalls:sys_exit_recvfrom / comm = = "java" & & @ start[ tid] / {$dur = (nsecs-@ start[ tid]) / 1000000; if ($dur > $1) {printf ("%-10u%-6d% 6d% s\ n", elapsed / 1000000, pid, $dur, @ query[ tid]);} delete (@ query[ tid]); delete (@ start[ tid]);}

Where comm represents the process name, tid represents the thread number, @ query [tid] is similar to @ start [tid] like map, with tid as key, this variable is like a thread-local variable.

By calling the script above, you can see that the execution time of each SQL is as follows:

$sudo BPFTRACE_STRLEN=80 bpftrace trace_slowsql_from_syscall.btAttaching 3 probes...Tracing jdbc SQL slower than 0 ms by sendto/recvfrom syscallTIME (ms) PID MS QUERY6398 3368243 125 select sleep 16427 3368243 22 select id from app_log al order by id desc limit 116431 3368243 20 select id,log_info,create_time,update_time,add_time from app_log where id=1169217492 3368243 21 select id,log_info,create_time,update_time Add_time from app_log where id=29214

Example: tracking slow SQL on server

Tracking SQL from the caller takes time, which includes network round-trip time. To get a more accurate SQL time, we can write a script to track the server-side mysql to observe the SQL time, as shown below:

Determine the executable files and entry functions of the mysqld service process

$which mysqld/usr/local/mysql/bin/mysqld# objdump can export the dynamic symbol table of the executable file. Do a few mysqld flame diagrams to find that dispatch_command is the entry function of SQL processing # in addition, because mysql is written by C++, the method name is rewritten by the compiler This is why * dispatch_command* fuzzy matching $objdump-tT / usr/local/mysql/bin/mysqld is used in the following script | grep dispatch_command00000000014efdf3 g F .text 0000000000252e _ Z16dispatch_commandP3THDPK8COM_DATA19enum_server_command00000000014efdf3 g DF .text 0000000000252e Base _ Z16dispatch_commandP3THDPK8COM_DATA19enum_server_command

Use uprobe to trace the call to dispatch_command, as follows:

#! / usr/bin/bpftraceBEGIN {printf ("Tracing mysqld SQL slower than% d ms. Ctrl-C to end.\ n ", $1); printf ("%-10s%-6s% 6s% s\ n "," TIME (ms) "," PID "," MS "," QUERY ");} uprobe:/usr/local/mysql/bin/mysqld:*dispatch_command* {if (arg2 = = (uint8) 3) {@ query [tid] = str (* arg1); @ start[ tid] = nsecs } uretprobe:/usr/local/mysql/bin/mysqld:*dispatch_command* / @ start [tid] / {$dur = (nsecs-@ start [tid]) / 1000000; if ($dur > $1) {printf ("%-10u%-6d% 6d% s\ n", elapsed / 1000000, pid, $dur, @ query [tid]);} delete (@ query [tid]); delete (@ start[ tid])};

The trace script is generally similar to the previous system call version, but the tracking point is different.

Example: find SQL that scans a large number of lines

It is well known that data is scanned when SQL is executed, and the more data is scanned, the worse SQL performance will be.

But for some intermediate cases, the number of SQL scan lines is neither more nor less, such as 2w. If these 2w pieces of data are in the cache, the SQL execution time will not be very long, resulting in not recorded in the slow query log, but if the amount of SQL concurrency is large, it will consume a lot of CPU.

For mysql, the scan line function is row_search_mvcc (if you often crawl the mysql stack, it is easy to find this function). It is called every scan line. If you trace this function in the trace script and record the number of calls, you can observe the scan lines of SQL, as shown below:

#! / usr/bin/bpftraceBEGIN {printf ("Tracing mysqld SQL scan row than% d. Ctrl-C to end.\ n", $1); printf ("%-10s%-6s% 6s s% s\ n", "TIME (ms)", "PID", "MS", "SCAN_NUM", "QUERY");} uprobe:/usr/local/mysql/bin/mysqld:*dispatch_command* {$COM_QUERY = (uint8) 3 If (arg2 = = $COM_QUERY) {@ query [tid] = str (* arg1); @ scan_ [tid] = nsecs;}} uprobe:/usr/local/mysql/bin/mysqld:*row_search_mvcc* {@ scan_ [tid] +;} uretprobe:/usr/local/mysql/bin/mysqld:*dispatch_command* / @ start [tid] / {$dur = (nsecs-@ start [tid]) / 1000000 If (@ scan_ Numtid] > $1) {printf ("%-10u%-6d% 6d% s\ n", elapsed / 1000000, pid, $dur, @ scan_num [tid], @ query[ tid]);} delete (@ queryNumtid]); delete (@ startNumtid]); delete (@ scan_ Num [tid]);}

The script works as follows:

Sudo BPFTRACE_STRLEN=80 bpftrace trace_mysql_scan.bt 200Attaching 4 probes...Tracing mysqld SQL scan row than 200. Ctrl-C to end.TIME (ms) PID MS SCAN_NUM QUERY150614 1892 4300 select * from app_log limit 300 # full table scan, slow! 17489 1892 424 43717 select count (*) from app_log # wide index scan, slow! 193013 1892 253 20000 select count (*) from app_log where id < 20000 # deep pagination, will query the first 20300, take the last 20300, slow! 213395 1892 1892 20300 select * from app_log limit 20000300 # index effect is not good, although only one piece of data will be found, but the amount of scanning data will not be small, slow! 430374 1892 1892 15000 select * from app_log where id < 20000 and seq = 15000 limit 1

As shown above, app_log is a test table I built with 43716 pieces of data, in which the id field is a self-incrementing primary key, and the seq value is the same as the id value, but there is no index.

You can see the above several scenarios, no matter what scenario, as long as the number of scan rows increases, the time will become longer, but none of them will exceed 500 milliseconds, because the table is very small and all the data can be cached in memory.

It can be seen that dynamic tracking tools like bpftrace are very powerful and more flexible than arthas. Arthas can only track a single function, while bpftrace can trace across functions.

These are all the contents of the article "what is the Linux dynamic tracking tool?" Thank you for reading! I believe you will gain a lot after reading this article. The editor will update different knowledge for you every day. If you want to learn more knowledge, please pay attention to the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.